voutcn / megahit

Ultra-fast and memory-efficient (meta-)genome assembler
http://www.ncbi.nlm.nih.gov/pubmed/25609793
GNU General Public License v3.0
585 stars 134 forks source link

Exit code 9. #335

Open schrad910 opened 2 years ago

schrad910 commented 2 years ago

Hi, I am having trouble with my code exiting out with error code 9. I've seen that means a system admin shutdown but I don't where to adjust the code so this now longer happens. Below is my log output: 2022-07-15 15:30:37 - MEGAHIT v1.2.9 2022-07-15 15:30:37 - Using megahit_core with POPCNT and BMI2 support 2022-07-15 15:30:37 - Convert reads to binary library 2022-07-15 15:30:37 - command /Users/nicoleschrad/opt/anaconda3/bin/megahit_core buildlib /Users/nicoleschrad/Mirror/USDA/Sequencing/novogene/usftp21.novogene.com/01.RawData/A/megahit_out/tmp/reads.lib /Users/nicoleschrad/Mirror/USDA/Sequencing/novogene/usftp21.novogene.com/01.RawData/A/megahit_out/tmp/reads.lib 2022-07-15 15:31:39 - b'INFO sequence/io/sequence_lib.cpp : 77 - Lib 0 (/Users/nicoleschrad/Mirror/USDA/Sequencing/novogene/usftp21.novogene.com/01.RawData/A/A_CKDN220037204-1A_H3G7TDSX5_L3_1.fq,/Users/nicoleschrad/Mirror/USDA/Sequencing/novogene/usftp21.novogene.com/01.RawData/A/A_CKDN220037204-1A_H3G7TDSX5_L3_2.fq): pe, 55731602 reads, 150 max length' 2022-07-15 15:31:39 - b'INFO utils/utils.h : 152 - Real: 61.6570\tuser: 62.9606\tsys: 10.6558\tmaxrss: 288763904' 2022-07-15 15:31:39 - k-max reset to: 127 2022-07-15 15:31:39 - Start assembly. Number of CPU threads 16 2022-07-15 15:31:39 - k list: 27,37,47,57,67,77,87,97,107,117,127 2022-07-15 15:31:39 - Memory used: 15461882265 2022-07-15 15:31:39 - Extracting solid (k+1)-mers and building sdbg for k = 27 2022-07-15 15:31:39 - command /Users/nicoleschrad/opt/anaconda3/bin/megahit_core read2sdbg -k 27 -m 1 --host_mem 15461882265 --mem_flag 1 --output_prefix /Users/nicoleschrad/Mirror/USDA/Sequencing/novogene/usftp21.novogene.com/01.RawData/A/megahit_out/tmp/k27/27 --num_cpu_threads 16 --read_lib_file /Users/nicoleschrad/Mirror/USDA/Sequencing/novogene/usftp21.novogene.com/01.RawData/A/megahit_out/tmp/reads.lib 2022-07-15 15:31:39 - b'INFO sequence/io/sequence_lib.cpp : 115 - Before reading, sizeof seq_package: 2535098896' 2022-07-15 15:31:48 - b'INFO sequence/io/sequence_lib.cpp : 117 - After reading, sizeof seq_package: 2535098896' 2022-07-15 15:31:48 - b'INFO sorting/read_to_sdbg_s1.cpp : 105 - 55731602 reads, 150 max read length, 8356984286 total bases' 2022-07-15 15:31:48 - b'INFO sorting/read_to_sdbg_s1.cpp : 109 - 2 words per substring' 2022-07-15 15:31:48 - b'INFO sorting/read_to_sdbg_s1.cpp : 124 - Number of files for mercy candidate reads: 8' 2022-07-15 15:31:48 - b'INFO sorting/base_engine.cpp : 148 - Preparing data...' 2022-07-15 15:31:48 - b'INFO sorting/read_to_sdbg_s2.cpp : 103 - 2 words per substring, words per dummy node ($v): 2' 2022-07-15 15:31:48 - b'INFO sorting/base_engine.cpp : 153 - Preparing data... Done. Time elapsed: 0.0063' 2022-07-15 15:31:48 - b'INFO sorting/base_engine.cpp : 156 - Preparing partitions and calculating bucket sizes...' 2022-07-15 15:32:01 - b'INFO sorting/base_engine.cpp : 80 - Minimum memory required: 3079757380 bytes' 2022-07-15 15:32:05 - b'INFO sorting/base_engine.cpp : 138 - Lv1 items: 1857074076, Lv2 items: 10201200' 2022-07-15 15:32:05 - b'INFO sorting/base_engine.cpp : 140 - Memory of derived class: 2600387864, Memory for Lv1+Lv2: 7509905904' 2022-07-15 15:32:05 - b'INFO sorting/base_engine.cpp : 168 - Preparing partitions and calculating bucket sizes... Done. Time elapsed: 17.5175' 2022-07-15 15:32:05 - b'INFO sorting/base_engine.cpp : 172 - Start main loop...' 2022-07-15 15:32:05 - b'INFO sorting/base_engine.cpp : 186 - Lv1 scanning from bucket 0 to 1096' 2022-07-15 15:32:27 - b'INFO sorting/base_engine.cpp : 193 - Lv1 scanning done. Large diff: 0. Time elapsed: 21.5042' 2022-07-15 15:33:07 - b'INFO sorting/base_engine.cpp : 199 - Lv1 fetching & sorting done. Time elapsed: 40.2038' 2022-07-15 15:33:07 - b'INFO sorting/base_engine.cpp : 186 - Lv1 scanning from bucket 1096 to 3275' 2022-07-15 15:33:29 - b'INFO sorting/base_engine.cpp : 193 - Lv1 scanning done. Large diff: 0. Time elapsed: 22.1955' 2022-07-15 15:34:13 - b'INFO sorting/base_engine.cpp : 199 - Lv1 fetching & sorting done. Time elapsed: 43.3323' 2022-07-15 15:34:13 - b'INFO sorting/base_engine.cpp : 186 - Lv1 scanning from bucket 3275 to 6632' 2022-07-15 15:34:35 - b'INFO sorting/base_engine.cpp : 193 - Lv1 scanning done. Large diff: 0. Time elapsed: 22.6714' 2022-07-15 15:35:22 - b'INFO sorting/base_engine.cpp : 199 - Lv1 fetching & sorting done. Time elapsed: 46.5448' 2022-07-15 15:35:22 - b'INFO sorting/base_engine.cpp : 186 - Lv1 scanning from bucket 6632 to 11454' 2022-07-15 15:35:46 - b'INFO sorting/base_engine.cpp : 193 - Lv1 scanning done. Large diff: 0. Time elapsed: 23.7014' 2022-07-15 15:36:35 - b'INFO sorting/base_engine.cpp : 199 - Lv1 fetching & sorting done. Time elapsed: 49.1569' 2022-07-15 15:36:35 - b'INFO sorting/base_engine.cpp : 186 - Lv1 scanning from bucket 11454 to 18339' 2022-07-15 15:37:00 - b'INFO sorting/base_engine.cpp : 193 - Lv1 scanning done. Large diff: 0. Time elapsed: 25.5917' 2022-07-15 15:37:52 - b'INFO sorting/base_engine.cpp : 199 - Lv1 fetching & sorting done. Time elapsed: 52.0603' 2022-07-15 15:37:52 - b'INFO sorting/base_engine.cpp : 186 - Lv1 scanning from bucket 18339 to 28498' 2022-07-15 15:38:20 - b'INFO sorting/base_engine.cpp : 193 - Lv1 scanning done. Large diff: 0. Time elapsed: 27.8127' 2022-07-15 15:39:28 - b'INFO sorting/base_engine.cpp : 199 - Lv1 fetching & sorting done. Time elapsed: 67.4313' 2022-07-15 15:39:28 - b'INFO sorting/base_engine.cpp : 186 - Lv1 scanning from bucket 28498 to 45257' 2022-07-15 15:39:57 - b'INFO sorting/base_engine.cpp : 193 - Lv1 scanning done. Large diff: 0. Time elapsed: 29.6739' 2022-07-15 15:40:55 - b'INFO sorting/base_engine.cpp : 199 - Lv1 fetching & sorting done. Time elapsed: 57.1754' 2022-07-15 15:40:55 - b'INFO sorting/base_engine.cpp : 186 - Lv1 scanning from bucket 45257 to 65536' 2022-07-15 15:41:16 - b'INFO sorting/base_engine.cpp : 193 - Lv1 scanning done. Large diff: 0. Time elapsed: 21.4666' 2022-07-15 15:41:49 - b'INFO sorting/base_engine.cpp : 199 - Lv1 fetching & sorting done. Time elapsed: 32.5511' 2022-07-15 15:41:49 - b'INFO sorting/base_engine.cpp : 204 - Main loop done. Time elapsed: 583.0755' 2022-07-15 15:41:49 - b'INFO sorting/base_engine.cpp : 207 - Postprocessing...' 2022-07-15 15:41:49 - b'INFO sorting/read_to_sdbg_s2.cpp : 618 - Number of $ A C G T A- C- G- T-:' 2022-07-15 15:41:49 - b'INFO sorting/read_to_sdbg_s2.cpp : 619 - 41843441 1219468311 2022188955 2026834675 1222869345 6576956 16615720 16569702 6549636' 2022-07-15 15:41:49 - b'INFO sorting/read_to_sdbg_s2.cpp : 624 - Total number of edges: 6579516741' 2022-07-15 15:41:49 - b'INFO sorting/read_to_sdbg_s2.cpp : 625 - Total number of ONEs: 6491361286' 2022-07-15 15:41:49 - b'INFO sorting/read_to_sdbg_s2.cpp : 627 - Total number of $v edges: 41843441' 2022-07-15 15:41:49 - b'INFO sorting/base_engine.cpp : 210 - Postprocess done. Time elapsed: 0.1974' 2022-07-15 15:41:49 - b'INFO utils/utils.h : 152 - Real: 610.4217\tuser: 9142.2209\tsys: 135.4512\tmaxrss: 10154659840' 2022-07-15 15:41:49 - Assemble contigs from SdBG for k = 27 2022-07-15 15:41:49 - command /Users/nicoleschrad/opt/anaconda3/bin/megahit_core assemble -s /Users/nicoleschrad/Mirror/USDA/Sequencing/novogene/usftp21.novogene.com/01.RawData/A/megahit_out/tmp/k27/27 -o /Users/nicoleschrad/Mirror/USDA/Sequencing/novogene/usftp21.novogene.com/01.RawData/A/megahit_out/intermediate_contigs/k27 -t 16 --min_standalone 300 --prune_level 2 --merge_len 20 --merge_similar 0.95 --cleaning_rounds 5 --disconnect_ratio 0.1 --low_local_ratio 0.2 --cleaning_rounds 5 --min_depth 2 --bubble_level 2 --max_tip_len -1 --careful_bubble 2022-07-15 15:44:34 - b'INFO main_assemble.cpp : 129 - Loading succinct de Bruijn graph: /Users/nicoleschrad/Mirror/USDA/Sequencing/novogene/usftp21.novogene.com/01.RawData/A/megahit_out/tmp/k27/27Done. Time elapsed: 164.934707' 2022-07-15 15:44:34 - b'INFO main_assemble.cpp : 133 - Number of Edges: 6579516741; K value: 27' 2022-07-15 15:44:34 - b'INFO main_assemble.cpp : 140 - Number of CPU threads: 16' 2022-07-15 15:46:28 - b'INFO assembly/sdbg_pruning.cpp : 160 - Removing tips with length less than 2; Accumulated tips removed: 898910; time elapsed: 8.1682' 2022-07-15 15:46:41 - b'INFO assembly/sdbg_pruning.cpp : 160 - Removing tips with length less than 4; Accumulated tips removed: 2492460; time elapsed: 13.3933' 2022-07-15 15:47:19 - b'INFO assembly/sdbg_pruning.cpp : 160 - Removing tips with length less than 8; Accumulated tips removed: 5452884; time elapsed: 37.3025' 2022-07-15 15:48:58 - b'INFO assembly/sdbg_pruning.cpp : 160 - Removing tips with length less than 16; Accumulated tips removed: 10872390; time elapsed: 98.9870' 2022-07-15 15:52:18 - b'INFO assembly/sdbg_pruning.cpp : 160 - Removing tips with length less than 32; Accumulated tips removed: 19258932; time elapsed: 200.4676' 2022-07-15 15:57:29 - b'INFO assembly/sdbg_pruning.cpp : 169 - Removing tips with length less than 54; Accumulated tips removed: 23978920; time elapsed: 311.0700' 2022-07-15 15:57:29 - b'INFO main_assemble.cpp : 158 - Tips removal done! Time elapsed(sec): 774.985' 2022-07-15 16:00:14 - Error occurs, please refer to /Users/nicoleschrad/Mirror/USDA/Sequencing/novogene/usftp21.novogene.com/01.RawData/A/megahit_out/log for detail 2022-07-15 16:00:14 - Command: /Users/nicoleschrad/opt/anaconda3/bin/megahit_core assemble -s /Users/nicoleschrad/Mirror/USDA/Sequencing/novogene/usftp21.novogene.com/01.RawData/A/megahit_out/tmp/k27/27 -o /Users/nicoleschrad/Mirror/USDA/Sequencing/novogene/usftp21.novogene.com/01.RawData/A/megahit_out/intermediate_contigs/k27 -t 16 --min_standalone 300 --prune_level 2 --merge_len 20 --merge_similar 0.95 --cleaning_rounds 5 --disconnect_ratio 0.1 --low_local_ratio 0.2 --cleaning_rounds 5 --min_depth 2 --bubble_level 2 --max_tip_len -1 --careful_bubble; Exit code -9

pguenzi-tiberi commented 1 year ago

https://github.com/voutcn/megahit/issues/272#issuecomment-643017856

I think it is a memory problem