fbreitwieser / krakenuniq

🐙 KrakenUniq: Metagenomics classifier with unique k-mer counting for more specific results
GNU General Public License v3.0
224 stars 43 forks source link

krakenuniq-build throws error jellyfish compacted_hash ErrorReading #66

Open joshua-theisen opened 4 years ago

joshua-theisen commented 4 years ago

I am trying to build a krakenuniq database using this code:

krakenuniq-download --threads 8 --db bactarch taxonomy
krakenuniq-download --threads 8 --dust --db bactarch refseq/bacteria refseq/archaea
krakenuniq-build --threads 8 --jellyfish-hash-size 64M --db bactarch

It works until the last step, when it runs for 3 hours and then throws this error:

terminate called after throwing an instance of 'jellyfish::compacted_hash::ErrorReading'
  what():  'database_880': File truncated
/PATH/TO/krakenuniq/build_db.sh: line 146: 43373 Aborted                 (core dumped) $JELLYFISH_BIN merge -o database.jdb.tmp database_*

I am running on a cluster with 512Gb of RAM and a small hash size, so memory shouldn't be an issue.

joshua-theisen commented 4 years ago

I've tried a number of different hash sizes; each throws a different error:

krakenuniq-download --threads 8 --db bactarch taxonomy
krakenuniq-download --threads 8 --dust --db bactarch refseq/bacteria refseq/archaea
krakenuniq-build --threads 8 --jellyfish-hash-size $jhs --db $db_path

where jhs is 64M, 128M, 640M, or 6400M, giving these errors:

##  - 64M error:
terminate called after throwing an instance of 'jellyfish::compacted_hash::ErrorReading'
  what():  'database_880': File truncated
/gpfs0/work/brian/scripts/krakenuniq/build_db.sh: line 146: 43373 Aborted                 (core dumped) $JELLYFISH_BIN merge -o database.jdb.tmp database_*
##  - 128M error:
db_sort: Getting database into memory ...db_sort: unable to mmap database.jdb: Cannot allocate memory
##  - 640M error:
db_sort: Getting database into memory ...db_sort: unable to mmap database.jdb: Cannot allocate memory
##  - 6400M error:
db_sort: Getting database into memory ...db_sort: unable to mmap database.jdb: Cannot allocate memory