Exit code -6

[microbiaki]

Hello,

I am using MEGAHIT v1.0.3-29-g707d683 and I got an error. Here is the part of the log that will be useful. Let me know if you need the entire file. (I got the error one and then I executed the same command with the --continue so it skipped the assembly of the previous k)

--- [Wed Jan 20 08:52:07 2016] Start assembly. Number of CPU threads 12 --- --- [Wed Jan 20 08:52:07 2016] Available memory: 1084374925312, used: 975937432780 --- [Wed Jan 20 08:52:07 2016] k list: 27,37,47,57,67,77,87 --- --- [Wed Jan 20 08:52:07 2016] Assembling contigs from SdBG for k = 77 --- cmd: /home/vedgcomb/megahit/megahit_asm_core assemble -s /home/vedgcomb/MEGAhit_test/20160119_Megahit_large_D2b237B/tmp/k77/77 -o /home/vedgcomb/MEGAhit_test/20160119_Megahit_large_D2b237B/intermediate_contigs/k77 -t 12 --min_standalone 260 --prune_level 2 --merge_len 20 --merge_similar 0.98 --low_local_ratio 0.2 --min_depth 2 --max_tip_len -1 --careful_bubble [assembler.cpp : 148] Loading succinct de Bruijn graph: /home/vedgcomb/MEGAhit_test/20160119_Megahit_large_D2b237B/tmp/k77/77 megahit_asm_core: succinct_dbg.cpp:672: void SuccinctDBG::LoadFromMultiFile(const char, bool): Assertion `tip_label_offset == num_tipnodes \ sdbg_reader.words_per_tip_label()' failed. Error occurs when assembling contigs for k = 77, please refer to /home/vedgcomb/MEGAhit_test/20160119_Megahit_large_D2b237B/log for detail [Exit code -6]

I really appreciate your insights.

Best, Maria

[voutcn]

Hi Maria,

Could you send me the log file, and the sdbg_info file tmp/k77/77.sdbg_info to me?

[microbiaki]

Hello Dinghua,

I rerun the analysis from scratch (only raising the numbers of threads) and now I got an error in kmer 37 which run fine the previous time (exactly the same input files)

MEGAHIT v1.0.3-29-g707d683 --- [Thu Jan 21 08:36:07 2016] Start assembly. Number of CPU threads 24 --- --- [Thu Jan 21 08:36:07 2016] Available memory: 1084374925312, used: 975937432780 --- [Thu Jan 21 08:36:07 2016] k list: 27,37,47,57,67,77,87 --- --- [Thu Jan 21 08:36:07 2016] Converting reads to binaries --- [read_lib_functions-inl.h : 209] Lib 0 (/home/vedgcomb/MEGAhit_test/D2b267B_R1_paired.fastq.gz,/home/vedgcomb/MEGAhit_test/D2b267B_R2_paired.fastq.gz): pe, 156528692 reads, 151 max length [utils.h : 124] Real: 290.7921 user: 261.9840 sys: 21.4120 maxrss: 165820 --- [Thu Jan 21 08:40:58 2016] Extracting solid (k+1)-mers for k = 27 --- --- [Thu Jan 21 08:53:13 2016] Building graph for k = 27 --- --- [Thu Jan 21 08:59:14 2016] Assembling contigs from SdBG for k = 27 --- --- [Thu Jan 21 09:10:20 2016] Local assembling k = 27 ---

--- [Thu Jan 21 09:22:42 2016] Extracting iterative edges from k = 27 to 37

--- [Thu Jan 21 09:38:04 2016] Building graph for k = 37 --- --- [Thu Jan 21 09:44:16 2016] Assembling contigs from SdBG for k = 37 --- Error occurs when assembling contigs for k = 37, please refer to /home/vedgcomb/MEGAhit_test/20160119_Megahit_large_D2b267B/log for detail [Exit code -6] vedgcomb@titan:~/me

I am attaching the log file and the temp/k37/37.sdbg_info of this current run

I really appreciate your help!

Best, Maria

On 21 January 2016 at 01:22, Dinghua Li notifications@github.com wrote:

Hi Maria,

Could you send me the log file, and the sdbg_info file tmp/k77/77.sdbg_info to me?

— Reply to this email directly or view it on GitHub https://github.com/voutcn/megahit/issues/68#issuecomment-173472636.

[voutcn]

Hi Maria,

Could you send the log & sdbg_info file to my email voutcn at gmail dot com?

[rjsorr]

just experienced the same problem on multiple runs with separate input files.

hope you can help? maybe just a memory issue, but its got 480gb to play with?

MEGAHIT v1.0.2 --- [Sun Jan 31 10:35:23 2016] Start assembly. Number of CPU threads 64 --- --- [Sun Jan 31 10:35:23 2016] k list: 21,31,41,51,61,71,81,91,99 --- --- [Sun Jan 31 10:35:23 2016] Converting reads to binaries --- /cluster/software/VERSIONS/megahit/v1.0.2/megahit_asm_core buildlib megahit_out/tmp/reads.lib megahit_out/tmp/reads.lib [read_lib_functions-inl.h : 209] Lib 0 (/projects/researchers/researchers01/rjsorr/genome/Breviates/Breviates_Hiseq2500/trimgalore/Orr4_Breviates_GTGAAA_L007_R1_001_val_1.fq.gz,/projects/researchers/researchers01/rjsorr/genome/Breviates/Breviates_Hiseq2500/trimgalore/Orr4_Breviates_GTGAAA_L007_R2_001_val_2.fq.gz): pe, 165585942 reads, 126 max length [read_lib_functions-inl.h : 209] Lib 1 (/projects/researchers/researchers01/rjsorr/genome/Breviates/Breviates_Hiseq2500/trimgalore/Orr4_Breviates_GTGAAA_L008_R1_001_val_1.fq.gz,/projects/researchers/researchers01/rjsorr/genome/Breviates/Breviates_Hiseq2500/trimgalore/Orr4_Breviates_GTGAAA_L008_R2_001_val_2.fq.gz): pe, 165449364 reads, 126 max length [read_lib_functions-inl.h : 209] Lib 2 (/projects/researchers/researchers01/rjsorr/genome/Breviates/Breviates_Hiseq2500/trimgalore/Orr-gDNA1-4-Breviates_GTGAAA_L008_R1_001_val_1.fq.gz,/projects/researchers/researchers01/rjsorr/genome/Breviates/Breviates_Hiseq2500/trimgalore/Orr-gDNA1-4-Breviates_GTGAAA_L008_R2_001_val_2.fq.gz): pe, 101893466 reads, 126 max length [read_lib_functions-inl.h : 209] Lib 3 (/projects/researchers/researchers01/rjsorr/genome/Breviates/pacbio/BreviatesPacbio/Breviates.consensus.fasta): se, 1090 reads, 1196534 max length [utils.h : 124] Real: 770.2152 user: 593.6458 sys: 41.1108 maxrss: 152196 --- [Sun Jan 31 10:48:14 2016] Extracting solid (k+1)-mers for k = 21 --- cmd: /cluster/software/VERSIONS/megahit/v1.0.2/megahit_sdbg_build count -k 21 -m 2 --host_mem 975839931187 --mem_flag 1 --gpu_mem 0 --output_prefix megahit_out/tmp/k21/21 --num_cpu_threads 64 --num_output_threads 21 --read_lib_file megahit_out/tmp/reads.lib [sdbg_builder.cpp : 114] Host memory to be used: 975839931187 [sdbg_builder.cpp : 115] Number CPU threads: 64 [cx1.h : 450] Preparing data... [read_lib_functions-inl.h : 253] Before reading, sizeof seq_package: 16981131328 [read_lib_functions-inl.h : 258] After reading, sizeof seq_package: 16981131328 [cx1_kmer_count.cpp : 104] 432929862 reads, 1196534 max read length [cx1.h : 457] Preparing data... Done. Time elapsed: 184.8318 [cx1.h : 464] Preparing partitions and initialing global data... [cx1_kmer_count.cpp : 195] 2 words per substring, 2 words per edge [cx1_kmer_count.cpp : 332] Memory for reads: 16981131328 [cx1_kmer_count.cpp : 333] max # lv.1 items = 5997625143 [cx1.h : 480] Preparing partitions and initialing global data... Done. Time elapsed: 65.9644 [cx1.h : 486] Start main loop... [cx1.h : 515] Lv1 scanning from bucket 0 to 753 [cx1.h : 528] Lv1 scanning done. Large diff: 26626431. Time elapsed: 60.1796 [cx1.h : 594] Lv1 fetching & sorting done. Time elapsed: 152.6450 [cx1.h : 515] Lv1 scanning from bucket 753 to 2948 [cx1.h : 528] Lv1 scanning done. Large diff: 297565662. Time elapsed: 425.0578 [cx1.h : 594] Lv1 fetching & sorting done. Time elapsed: 234.4000 [cx1.h : 515] Lv1 scanning from bucket 2948 to 6473 [cx1.h : 528] Lv1 scanning done. Large diff: 812887942. Time elapsed: 1184.2363 [cx1.h : 594] Lv1 fetching & sorting done. Time elapsed: 256.7940 [cx1.h : 515] Lv1 scanning from bucket 6473 to 11432 [cx1.h : 528] Lv1 scanning done. Large diff: 1415402206. Time elapsed: 2128.8859 [cx1.h : 594] Lv1 fetching & sorting done. Time elapsed: 259.0757 [cx1.h : 515] Lv1 scanning from bucket 11432 to 18135 [cx1.h : 528] Lv1 scanning done. Large diff: 2066125865. Time elapsed: 3229.8806 [cx1.h : 594] Lv1 fetching & sorting done. Time elapsed: 253.2128 [cx1.h : 515] Lv1 scanning from bucket 18135 to 27427 [cx1.h : 528] Lv1 scanning done. Large diff: 2819322187. Time elapsed: 4457.8069 megahit_sdbg_build: edge_io.h:92: void EdgeWriter::write(uint32_t*, int32_t, int): Assertion `prec[bucket].thread_id == -1' failed. Error occurs when running "sdbg_builder count/read2sdbg", please refer to megahit_out/log for detail [Exit code -6]

[microbiaki]

I dropped the ball on this because I was too busy the last week.

I got the error when I was running it in the institution's cluster, where I have 1T of RAM and 1T of disk space. Dinghua suggested me to try the dataset in "https://github.com/voutcn/megahit/wiki/An-example-of-real-assembly?" but meanwhile I installed megahit on an Ubuntu virtual box in the lab's desktop (64G RAM; 100G of disk space) and it worked fine with the same datasets that were creating the error in the cluster.

[voutcn]

@mpachiad I guess it is a platform related problem, related to hardward atomic operations.

@rjsorr The error you came across is different from that of @mpachiad . You are putting long sequences into MEGAHIT, which makes the graph construction halt. I will fix this issue in one week.

[rjsorr]

cheers @voutcn. great if you can come with a solution. Actually this is pacbio reads assembled using canu, so 1mb+ contigs. This best would be if I could submit pacbio raw (filtered) reads with the -r option, but Megahit does not seem to deal with this so well, hence the assembly of pacbio before assembling it with the PE reads.

regards

[hcwang24]

@voutcn Hi, I just started compiling and testing my first sample using MegaHit. However, something seems wrong when I start the program. Here's what's in the log file.

MEGAHIT v1.1.0-pre --- [Sat Feb 13 16:41:28 2016] Start assembly. Number of CPU threads 24 --- --- [Sat Feb 13 16:41:28 2016] Available memory: 25323257856, used: 22790932070 --- [Sat Feb 13 16:41:28 2016] k list: 21,41,61,81,99 --- --- [Sat Feb 13 16:41:28 2016] Converting reads to binaries --- /lustre/home/hcwang/data/megahit-master/megahit_asm_core buildlib /home/hcwang/sample/S9_megahit_output/tmp/reads.lib /home/hcwang/sample/S9_megahit_output/tmp/reads.lib megahit_asm_core: sequence_manager.cpp:121: int64_t SequenceManager::ReadShortReads(int64_t, int64_t, bool, bool, bool): Assertion `kseq_read(kseqreaders[1]) >= 0' failed. Error occurs when running "megahit_asm_core buildlib"; please refer to /home/hcwang/sample/S9_megahit_output/S9_megahit.log for detail [Exit code -6]

[voutcn]

@wanghanchenwhc Could you please show your command to run Megahit here? It seems that you are using incorrect paired-end files.

[voutcn]

@rjsorr Sorry I am a bit busy this week, I will push a new commit ASAP

[hcwang24]

@voutcn I'm running them on UNIX server: export megahit_dir=echo $R1file | sed "s/R1_processed_sequence.fasta/megahit_output/g"; export megahit_tmp=echo $R1file | sed "s/\/home\/hcwang\/sample\///g"; export megahit_out=echo $megahit_tmp | sed "s/R1_processed_sequence.fasta/megahit/g";

These three lines just to set the file names

/global/software/megahit/megahit-1.1.0-pre/megahit --presets meta -1 $R1file -2 $R2file -o $megahit_dir --out-prefix $megahit_out;

I'm suspecting that could it be problematic when I try to run FASTQ-converted FASTA files instead of using the original FASTQ files?

[jvollme]

I recently encountered this eror code as well. I am not using any long reads and Megahit used to work perfectly before on this exact server. I am using MEGAHIT v1.0.1. I gave it 390 Gb of RAM of which it used only a little fraction (peak RAM usage monitored by our SGE queuing system). Heres the last portion of my output:

--- [Wed Mar 9 20:46:27 2016] Extracting solid (k+1)-mers for k = 27 --- --- [Wed Mar 9 23:00:49 2016] Building graph for k = 27 --- --- [Thu Mar 10 02:03:53 2016] Assembling contigs from SdBG for k = 27 --- --- [Thu Mar 10 04:07:17 2016] Local assembling k = 27 --- --- [Thu Mar 10 06:10:29 2016] Extracting iterative edges from k = 27 to 47 --- --- [Thu Mar 10 06:45:08 2016] Building graph for k = 47 --- --- [Thu Mar 10 07:34:39 2016] Assembling contigs from SdBG for k = 47 --- --- [Thu Mar 10 09:01:14 2016] Local assembling k = 47 --- --- [Thu Mar 10 10:26:13 2016] Extracting iterative edges from k = 47 to 67 --- --- [Thu Mar 10 10:50:47 2016] Building graph for k = 67 --- --- [Thu Mar 10 11:27:42 2016] Assembling contigs from SdBG for k = 67 --- --- [Thu Mar 10 13:46:13 2016] Local assembling k = 67 --- --- [Thu Mar 10 14:24:07 2016] Extracting iterative edges from k = 67 to 87 --- Traceback (most recent call last): File "/usr/lib64/python2.7/logging/init.py", line 875, in emit self.flush() File "/usr/lib64/python2.7/logging/init.py", line 835, in flush self.stream.flush() IOError: [Errno 5] Input/output error Logged from file megahit, line 832 Error occurs when running iterator for k = 67 to k = 87, please refer to ./megahit_out/log for detail [Exit code -6]

I restarted the assembly (using "--continue") in the vague hope that this error just does not happen again but do not have much hope in that.

[voutcn]

@jvollme -6 was returned by an assertion failure, the log file should have recorded that failure. So even though this thread is talking about "-6", the problems here are not the same. Would you open a new issue and paste the full contents of log file there?

[Young331]

I encountered this eror code as well. I am using MEGAHIT v1.2.1-beta. hope you can help? The log file is following: MEGAHIT v1.2.1-beta --- [Tue Apr 9 14:58:18 2019] Start assembly. Number of CPU threads 68 --- --- [Tue Apr 9 14:58:18 2019] Available memory: 135082373120, used: 121574135808 --- [Tue Apr 9 14:58:18 2019] Converting reads to binary library --- /scale_wlg_persistent/filesets/project/uoa02698/modules/MEGAHIT-1.2.1-beta-Linux-static/bin/megahit_core_no_hw_accel buildlib assembly_1117/tmp/reads.lib assembly_1117/tmp/reads.lib [INFO src/main_buildlib.cpp 122] Lib 0 (/scale_wlg_persistent/filesets/project/uoa02698/DNA-1117_1.clean.fq,/scale_wlg_persistent/filesets/project/uoa02698/DNA-1117_2.clean.fq): pe, 531284512 reads, 150 max length [INFO src/main_buildlib.cpp 119] Real: 636.1904 user: 554.7897 sys: 51.2243 maxrss: 162060 --- [Tue Apr 9 15:08:54 2019] k list: 79,89,99,109,119,129,139,149 --- --- [Tue Apr 9 15:08:54 2019] Extracting solid (k+1)-mers for k = 79 --- cmd: /scale_wlg_persistent/filesets/project/uoa02698/modules/MEGAHIT-1.2.1-beta-Linux-static/bin/megahit_core_no_hw_accel count -k 79 -m 2 --host_mem 121574135808 --mem_flag 1 --output_prefix assembly_1117/tmp/k79/79 --num_cpu_threads 68 --num_output_threads 22 --read_lib_file assembly_1117/tmp/reads.lib [INFO src/main_sdbg_build.cpp 91] Host memory to be used: 121574135808 [INFO src/main_sdbg_build.cpp 92] Number CPU threads: 68 [INFO src/main_sdbg_build.cpp 264] Preparing data... terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Error occurs when running "sdbg_builder count/read2sdbg",please refer to assembly_1117/log for detail [Exit code -6]