search

DaehwanKimLab / centrifuge

Classifier for metagenomic sequences
GNU General Public License v3.0
246 stars 73 forks source link

NT Databe failure (ERR): centrifuge-class died with signal 9 (KILL) #224

Open JochenSchaefergmxde opened 2 years ago

JochenSchaefergmxde commented 2 years ago

/mnt/test# centrifuge -p 64 --verbose -x centrifuge/nt --met-file metric.txt --report-file ln_nt_centrifuge_report.txt -S ln_nt_centrifuge_results.txt /mnt/test/fastq.fastq (INFO): Before arg handling: (INFO): Wrapper args: [ ] (INFO): Binary args: [ -p 64 --verbose -x centrifuge/nt --met-file metric.txt --report-file ln_nt_centrifuge_report.txt -S ln_nt_centrifuge_results.txt /mnt/test/fastq.fastq ] (INFO): After arg handling: (INFO): Binary args: [ -p 64 --verbose -x centrifuge/nt --met-file metric.txt --report-file ln_nt_centrifuge_report.txt -S ln_nt_centrifuge_results.txt /mnt/test/fastq.fastq ] (INFO): /usr/local/bin/centrifuge-class --wrapper basic-0 -p 64 --verbose -x centrifuge/nt --met-file metric.txt --report-file ln_nt_centrifuge_report.txt -S ln_nt_centrifuge_results.txt /mnt/test/fastq.fastq Applying preset: 'sensitive' using preset menu 'V0' Final policy string: 'SEED=0,22;DPS=15;ROUNDS=2;IVAL=S,1,1.15' Input bt2 file: "centrifuge/nt" Query inputs (DNA, FASTQ): /mnt/test/fastq.fastq Quality inputs: Output file: "ln_nt_centrifuge_results.txt" Local endianness: little Sanity checking: disabled Assertions: disabled Entered driver(): 20:19:28 Creating PatternSource: 20:19:28 Opening hit output file: 20:19:28 About to initialize fw Ebwt: 20:19:28 Trying centrifuge/nt About to open input files: 20:19:28 Opening "centrifuge/nt.1.cf" Opening "centrifuge/nt.2.cf" Finished opening input files: 20:19:28 Reading header: 20:19:28 Headers: len: 161320546158 bwtLen: 161320546159 sz: 40330136540 bwtSz: 40330136540 lineRate: 7 offRate: 4 offMask: 0xfffffffffffffff0 ftabChars: 14 eftabLen: 28 eftabSz: 224 ftabLen: 268435457 ftabSz: 2147483656 offsLen: 10082534135 offsSz: 80660273080 lineSz: 128 sideSz: 128 sideBwtSz: 96 sideBwtLen: 384 numSides: 420105589 numLines: 420105589 ebwtTotLen: 53773515392 ebwtTotSz: 53773515392 color: 0 reverse: 0 Reading plen (46203723): 20:19:28 Opening "centrifuge/nt.3.cf" Opening "centrifuge/nt.4.cf" About to open input files: 20:20:28 Opening "centrifuge/nt.1.cf" Opening "centrifuge/nt.2.cf" Finished opening input files: 20:20:28 Reading header: 20:20:28 Reading plen (46203723): 20:20:28 Reading rstarts (777423930): 20:20:28 Reading ebwt (53773515392): 20:20:32 Reading fchr (5) Reading ftab (268435457): 20:21:11 Reading eftab (28): 20:21:12 Reading offs (10082534135 64-bit words): 20:22:35 Headers: len: 161320546158 bwtLen: 161320546159 sz: 40330136540 bwtSz: 40330136540 lineRate: 7 offRate: 4 offMask: 0xfffffffffffffff0 ftabChars: 14 eftabLen: 28 eftabSz: 224 ftabLen: 268435457 ftabSz: 2147483656 offsLen: 10082534135 offsSz: 80660273080 lineSz: 128 sideSz: 128 sideBwtSz: 96 sideBwtLen: 384 numSides: 420105589 numLines: 420105589 ebwtTotLen: 53773515392 ebwtTotSz: 53773515392 color: 0 reverse: 0 creating patternsource for 1-th input: 20:24:24 Dispatching to search driver: 20:24:24 (ERR): centrifuge-class died with signal 9 (KILL)

mourisl commented 2 years ago

What is the memory size of the server you are using?

JochenSchaefergmxde commented 2 years ago

Aws instance with 200 Gb even with m notebook 32 GB the same with NT database ! Kraken2 is working with 160gb database perfectly.Am 29.11.2021 22:12 schrieb Li Song @.***>: What is the memory size of the server you are using?

—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

mourisl commented 2 years ago

32GB is not enough for nt database, but 200GB should be enough. The memory requirement is about the same size as the index file size.

I think the crash happens after the index was loaded so the memory might not be the cause. Which centrifuge version did you use? Could you please show me the first few lines of the fastq.fastq file? Could you please also run centrifuge without --met-file option? Thank you.

khyox commented 2 years ago

In case this helps provide recent context on nt, my last run of Centrifuge classifier using a current NCBI BLAST nt database (Centrifuge index with size 327 GB) required 650 GB of main memory when using 256 threads (128 cores).

JochenSchaefergmxde commented 2 years ago

In case this helps provide recent context on nt, my last run of Centrifuge classifier using a current NCBI BLAST nt database (Centrifuge index with size 327 GB) required 650 GB of main memory when using 256 threads (128 cores).

That cant be the Solution ! If not enough memory is available, Centrifuge must work longer but not an error ! There must be something wrong.

JochenSchaefergmxde commented 2 years ago

32GB is not enough for nt database, but 200GB should be enough. The memory requirement is about the same size as the index file size.

I think the crash happens after the index was loaded so the memory might not be the cause. Which centrifuge version did you use? Could you please show me the first few lines of the fastq.fastq file? Could you please also run centrifuge without --met-file option? Thank you.

I try it with different sources of Fastq files (Nanopore/Dantalabs) and i get the same error. The other error is that the metric file is not written (and it doesn't matter with or without i add it to see more why ) and the Version Number is not printet with Verbose out put ! ;) But ist is the newest directly from github.