search

voutcn / megahit

Ultra-fast and memory-efficient (meta-)genome assembler
http://www.ncbi.nlm.nih.gov/pubmed/25609793
GNU General Public License v3.0
585 stars 134 forks source link

Assembly killed exit code 9 despite sufficient memory? #327

Closed jzrapp closed 2 years ago

jzrapp commented 2 years ago

Hi @voutcn, I'm trying to run an assembly via a slurm cluster submission.

I requested 500 GB RAM and 24 cores.

The log files indicates 

2022-02-16 14:09:57 - Memory used: 419246254489
2022-02-16 14:09:57 - Extract solid (k+1)-mers for k = 21 
2022-02-16 14:09:57 - command /prg/miniconda/3/envs/anvio-7.1/bin/megahit_core count -k 21 -m 2 --host_mem 419246254489 --mem_flag 1 --output_prefix /project/02_FASTA/Thores2018_TEMP/tmp/k21/21 --num_cpu_threads 24 --read_lib_file /project/02_FASTA/Thores2018_TEMP/tmp/reads.lib
2022-02-16 14:09:57 - b'INFO  sorting/base_engine.cpp       :  148 - Preparing data...'
2022-02-16 14:09:57 - b'INFO  sequence/io/sequence_lib.cpp  :  115 - Before reading, sizeof seq_package: 9591240656'
2022-02-16 14:10:38 - b'INFO  sequence/io/sequence_lib.cpp  :  117 - After reading, sizeof seq_package: 9591240656'
2022-02-16 14:10:39 - b'INFO  sorting/kmer_counter.cpp      :   76 - 240632210 reads, 151 max read length'
2022-02-16 14:10:39 - b'INFO  sorting/kmer_counter.cpp      :   82 - 2 words per substring, 2 words per edge'
2022-02-16 14:10:56 - b'INFO  sorting/base_engine.cpp       :  153 - Preparing data... Done. Time elapsed: 59.0315'
2022-02-16 14:10:56 - b'INFO  sorting/base_engine.cpp       :  156 - Preparing partitions and calculating bucket sizes...'
2022-02-16 14:11:39 - b'INFO  sorting/base_engine.cpp       :   80 - Minimum memory required: 12824284264 bytes'
2022-02-16 14:11:48 - Error occurs, please refer to /project/02_FASTA/Thores2018_TEMP/log for detail
2022-02-16 14:11:48 - Command: /prg/miniconda/3/envs/anvio-7.1/bin/megahit_core count -k 21 -m 2 --host_mem 419246254489 --mem_flag 1 --output_prefix /project/02_FASTA/Thores2018_TEMP/tmp/k21/21 --num_cpu_threads 24 --read_lib_file /project/02_FASTA/Thores2018_TEMP/tmp/reads.lib; Exit code -9

To me it looks like the minimum memory required is far below what I have available.. could you tell me where the problem is if it's not related to memory?

Thanks! Josephine

jzrapp commented 2 years ago

I realized that the issue was not with megahit, but with the snakemake to slurm command line that submitted the job to our cluster system. My apologies!