Closed ch8316f5eyu closed 4 years ago
Hi,
this could be the few core dumps during the gap reassembly mentioned on the github CSA start page:
"You may see a few "core dumps", these are single gap re-assembly jobs that failed, you should just not care. Some gap re-assembly jobs may run very long due to huge read pile ups in that location, therefore we force gap re-assemblies to stop after 20 minutes using "timeout"."
You can just keep the tool running, it should finish with good results. If the pipeline really stops, post a list (ls -lhtr) of the 03_.... directory here, please. Also post the commandline you used to start CSA and the full log file
What type of reads are you using (technology/N50 readlength/coverage)?
And yes, 188Gb RAM is pretty low for a mammalian genome assembly, but I would expect a crash or infinite runtime earlier in the pipeline! I think you may need about 512Gb. We wrote this in the publication and on the github CSA start page:
"We show that CSA is able to produce chromosomal-level assemblies for smaller vertebrate genomes (fishes, birds) within 12 h on low-cost computing equipment ($1,000–2,000, Intel i7, 128 GB RAM), using just long-read data and a diverged reference genome (divergence time ∼65 million years ago [Mya]) as input. Larger mammalian genomes, such as human, can be assembled within 16 h on server equipment (Intel Xeon, 1 TB RAM)."
There is the command line:
~/app/CSA2.6/CSA2.6c.pl -r fastq/terc_Control.fa.gz -g ../../Reference/Genome_data/mm10/mm10.fa -t 10 -o N33 -d CSA > script/CAS.sh
There is the output massage:
`Sat Jun 20 07:27:21 CST 2020
RUN WTDBG on ../../fastq/terc_Control.fa.gz
Sat Jun 20 09:18:50 CST 2020 CREATE CONSENSUS / use wtdbg-cns
Sat Jun 20 12:12:18 CST 2020 COMPARE ASSEMBLIES AND SPLIT CONTIGS AT DIFFERENCES
Sat Jun 20 12:32:11 CST 2020 ASSEMBLY IMPROVEMENTS BY READ RE-MAPPING AND SPLITTING
PRIMARY CONTIG STATS: Total: 2422821200 Count: 7188 Average: 337064.72 Median: 72201 N00: 13324505 1 N10: 4195572 45 N20: 3071029 112 N30: 2312588 203 N40: 1751953 324 N50: 1279915 489 N60: 919184 714 N70: 650348 1027 N80: 413460 1489 N90: 188768 2346 N100: 2011 7188
Sat Jun 20 13:16:46 CST 2020 FINISHED CSA STEP1 real 349m25.456s user 3062m38.996s sys 76m49.160s
Sat Jun 20 13:16:47 CST 2020 MASKING REFERENCE BY SELF-ALIGNMENT
Sat Jun 20 14:45:43 CST 2020 CREATE LAST DATABASE FOR WHOLE GENOME ALIGNMENT
Sat Jun 20 15:01:03 CST 2020 RUN WHOLE GENOME ALIGNMENT BY LAST
Sat Jun 20 16:56:09 CST 2020 RUN RAGOUT TO ORDER CONTIGS ACCORDING TO REFERENCE
PRIMARY SCAFFOLD STATS: Total: 2506267816 Count: 1458 Average: 1718976.55 Median: 13842 N00: 188292983 1 N10: 174131707 2 N20: 166547346 3 N30: 146728522 5 N40: 143279141 7 N50: 125013353 9 N60: 119070898 11 N70: 111949287 13 N80: 106031649 15 N90: 88708220 18 N100: 2011 1458
Sat Jun 20 17:04:33 CST 2020 FINISHED CSA STEP2
real 227m45.360s user 1036m24.456s sys 138m14.948s
Sat Jun 20 17:04:56 CST 2020 MAP RAW READS TO ASSEMBLY
Sat Jun 20 17:22:07 CST 2020 WRITE GAP/CTGEND MATCHING RAW READS TO FILES
Sat Jun 20 17:25:35 CST 2020 RUN LOCAL ASSEMBLIES
Error in `/home/yu/app/CSA2.6/INSTALL/../bin/wtdbg2.2/wtdbg-cns': free(): corrupted unsorted chunks: 0x0000000000ca4da0
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fb2657ca7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7fb2657d337a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fb2657d753c]
/home/yu/app/CSA2.6/INSTALL/../bin/wtdbg2.2/wtdbg-cns[0x404415]
/home/yu/app/CSA2.6/INSTALL/../bin/wtdbg2.2/wtdbg-cns[0x4036e1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fb265d3e6ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fb26585a41d]
======= Memory map: ========
00400000-00421000 r-xp 00000000 08:41 2139283 /home/yu/app/CSA2.6/bin/wtdbg2.2/wtdbg-cns
00620000-00621000 r--p 00020000 08:41 2139283 /home/yu/app/CSA2.6/bin/wtdbg2.2/wtdbg-cns
00621000-00622000 rw-p 00021000 08:41 2139283 /home/yu/app/CSA2.6/bin/wtdbg2.2/wtdbg-cns
00ca4000-00d6f000 rw-p 00000000 00:00 0 [heap]
7fb248000000-7fb248023000 rw-p 00000000 00:00 0
7fb248023000-7fb24c000000 ---p 00000000 00:00 0
7fb250000000-7fb25002c000 rw-p 00000000 00:00 0
7fb25002c000-7fb254000000 ---p 00000000 00:00 0
7fb2557fb000-7fb2557fc000 ---p 00000000 00:00 0
7fb2557fc000-7fb255ffc000 rw-p 00000000 00:00 0
7fb258000000-7fb258023000 rw-p 00000000 00:00 0
7fb258023000-7fb25c000000 ---p 00000000 00:00 0
7fb25d878000-7fb25dc79000 rw-p 00000000 00:00 0
7fb25deee000-7fb25df7b000 rw-p 00000000 00:00 0
7fb25df7b000-7fb25dffc000 rw-p 00000000 00:00 0
7fb25dffc000-7fb25dffd000 ---p 00000000 00:00 0
7fb25dffd000-7fb25e7fd000 rw-p 00000000 00:00 0
7fb25e7fd000-7fb25e7fe000 ---p 00000000 00:00 0
7fb25e7fe000-7fb25effe000 rw-p 00000000 00:00 0
7fb25effe000-7fb25efff000 ---p 00000000 00:00 0
7fb25efff000-7fb25f7ff000 rw-p 00000000 00:00 0
7fb25f7ff000-7fb25f800000 ---p 00000000 00:00 0
7fb25f800000-7fb260000000 rw-p 00000000 00:00 0
7fb260000000-7fb260021000 rw-p 00000000 00:00 0
7fb260021000-7fb264000000 ---p 00000000 00:00 0
7fb26404a000-7fb26404b000 ---p 00000000 00:00 0
7fb26404b000-7fb26484b000 rw-p 00000000 00:00 0
7fb264d3a000-7fb264d51000 r-xp 00000000 08:41 8127216 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fb264d51000-7fb264f50000 ---p 00017000 08:41 8127216 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fb264f50000-7fb264f51000 r--p 00016000 08:41 8127216 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fb264f51000-7fb264f52000 rw-p 00017000 08:41 8127216 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fb264f52000-7fb264f53000 ---p 00000000 00:00 0
7fb264f53000-7fb265753000 rw-p 00000000 00:00 0
7fb265753000-7fb265913000 r-xp 00000000 08:41 8133143 /lib/x86_64-linux-gnu/libc-2.23.so
7fb265913000-7fb265b13000 ---p 001c0000 08:41 8133143 /lib/x86_64-linux-gnu/libc-2.23.so
7fb265b13000-7fb265b17000 r--p 001c0000 08:41 8133143 /lib/x86_64-linux-gnu/libc-2.23.so
7fb265b17000-7fb265b19000 rw-p 001c4000 08:41 8133143 /lib/x86_64-linux-gnu/libc-2.23.so
7fb265b19000-7fb265b1d000 rw-p 00000000 00:00 0
7fb265b1d000-7fb265b36000 r-xp 00000000 08:41 8126974 /lib/x86_64-linux-gnu/libz.so.1.2.8
7fb265b36000-7fb265d35000 ---p 00019000 08:41 8126974 /lib/x86_64-linux-gnu/libz.so.1.2.8
7fb265d35000-7fb265d36000 r--p 00018000 08:41 8126974 /lib/x86_64-linux-gnu/libz.so.1.2.8
7fb265d36000-7fb265d37000 rw-p 00019000 08:41 8126974 /lib/x86_64-linux-gnu/libz.so.1.2.8
7fb265d37000-7fb265d4f000 r-xp 00000000 08:41 8133127 /lib/x86_64-linux-gnu/libpthread-2.23.so
7fb265d4f000-7fb265f4e000 ---p 00018000 08:41 8133127 /lib/x86_64-linux-gnu/libpthread-2.23.so
7fb265f4e000-7fb265f4f000 r--p 00017000 08:41 8133127 /lib/x86_64-linux-gnu/libpthread-2.23.so
7fb265f4f000-7fb265f50000 rw-p 00018000 08:41 8133127 /lib/x86_64-linux-gnu/libpthread-2.23.so
7fb265f50000-7fb265f54000 rw-p 00000000 00:00 0
7fb265f54000-7fb265f5b000 r-xp 00000000 08:41 8130839 /lib/x86_64-linux-gnu/librt-2.23.so
7fb265f5b000-7fb26615a000 ---p 00007000 08:41 8130839 /lib/x86_64-linux-gnu/librt-2.23.so
7fb26615a000-7fb26615b000 r--p 00006000 08:41 8130839 /lib/x86_64-linux-gnu/librt-2.23.so
7fb26615b000-7fb26615c000 rw-p 00007000 08:41 8130839 /lib/x86_64-linux-gnu/librt-2.23.so
7fb26615c000-7fb266264000 r-xp 00000000 08:41 8133146 /lib/x86_64-linux-gnu/libm-2.23.so
7fb266264000-7fb266463000 ---p 00108000 08:41 8133146 /lib/x86_64-linux-gnu/libm-2.23.so
7fb266463000-7fb266464000 r--p 00107000 08:41 8133146 /lib/x86_64-linux-gnu/libm-2.23.so
7fb266464000-7fb266465000 rw-p 00108000 08:41 8133146 /lib/x86_64-linux-gnu/libm-2.23.so
7fb266465000-7fb26648b000 r-xp 00000000 08:41 8133125 /lib/x86_64-linux-gnu/ld-2.23.so
7fb2664b8000-7fb2664dc000 rw-p 00000000 00:00 0
7fb2664fd000-7fb26653e000 rw-p 00000000 00:00 0
7fb26663f000-7fb266665000 rw-p 00000000 00:00 0
7fb266668000-7fb26668a000 rw-p 00000000 00:00 0
7fb26668a000-7fb26668b000 r--p 00025000 08:41 8133125 /lib/x86_64-linux-gnu/ld-2.23.so
7fb26668b000-7fb26668c000 rw-p 00026000 08:41 8133125 /lib/x86_64-linux-gnu/ld-2.23.so
7fb26668c000-7fb26668d000 rw-p 00000000 00:00 0
7ffe3c556000-7ffe3c578000 rw-p 00000000 00:00 0 [stack]
7ffe3c5c3000-7ffe3c5c5000 r--p 00000000 00:00 0 [vvar]
7ffe3c5c5000-7ffe3c5c7000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Error in `/home/yu/app/CSA2.6/INSTALL/../bin/wtdbg2.2/wtdbg-cns': free(): corrupted unsorted chunks: 0x00000000010c7da0
====== Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f4be27617e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f4be276a37a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f4be276e53c]
/home/yu/app/CSA2.6/INSTALL/../bin/wtdbg2.2/wtdbg-cns[0x404415]
/home/yu/app/CSA2.6/INSTALL/../bin/wtdbg2.2/wtdbg-cns[0x4036e1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f4be2cd56ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f4be27f141d]
======= Memory map: ========
00400000-00421000 r-xp 00000000 08:41 2139283 /home/yu/app/CSA2.6/bin/wtdbg2.2/wtdbg-cns
00620000-00621000 r--p 00020000 08:41 2139283 /home/yu/app/CSA2.6/bin/wtdbg2.2/wtdbg-cns
00621000-00622000 rw-p 00021000 08:41 2139283 /home/yu/app/CSA2.6/bin/wtdbg2.2/wtdbg-cns
010c7000-01175000 rw-p 00000000 00:00 0 [heap]
7f4bcc000000-7f4bcc023000 rw-p 00000000 00:00 0
7f4bcc023000-7f4bd0000000 ---p 00000000 00:00 0
7f4bd0000000-7f4bd002c000 rw-p 00000000 00:00 0
7f4bd002c000-7f4bd4000000 ---p 00000000 00:00 0
7f4bd4000000-7f4bd4023000 rw-p 00000000 00:00 0
7f4bd4023000-7f4bd8000000 ---p 00000000 00:00 0
7f4bd9ffc000-7f4bd9ffd000 ---p 00000000 00:00 0
7f4bd9ffd000-7f4bda7fd000 rw-p 00000000 00:00 0
7f4bda7fd000-7f4bda7fe000 ---p 00000000 00:00 0
7f4bda7fe000-7f4bdaffe000 rw-p 00000000 00:00 0
7f4bdaffe000-7f4bdafff000 ---p 00000000 00:00 0
7f4bdafff000-7f4bdb7ff000 rw-p 00000000 00:00 0
7f4bdb7ff000-7f4bdb800000 ---p 00000000 00:00 0
7f4bdb800000-7f4bdc000000 rw-p 00000000 00:00 0
7f4bdc000000-7f4bdc021000 rw-p 00000000 00:00 0
7f4bdc021000-7f4be0000000 ---p 00000000 00:00 0
7f4be0416000-7f4be0617000 rw-p 00000000 00:00 0
7f4be0617000-7f4be06a4000 rw-p 00000000 00:00 0
7f4be071e000-7f4be075f000 rw-p 00000000 00:00 0
7f4be075f000-7f4be07e0000 rw-p 00000000 00:00 0
7f4be07e0000-7f4be07e1000 ---p 00000000 00:00 0
7f4be07e1000-7f4be0fe1000 rw-p 00000000 00:00 0
7f4be0fe1000-7f4be0fe2000 ---p 00000000 00:00 0
7f4be0fe2000-7f4be17e2000 rw-p 00000000 00:00 0
7f4be1cd1000-7f4be1ce8000 r-xp 00000000 08:41 8127216 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f4be1ce8000-7f4be1ee7000 ---p 00017000 08:41 8127216 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f4be1ee7000-7f4be1ee8000 r--p 00016000 08:41 8127216 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f4be1ee8000-7f4be1ee9000 rw-p 00017000 08:41 8127216 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f4be1ee9000-7f4be1eea000 ---p 00000000 00:00 0
7f4be1eea000-7f4be26ea000 rw-p 00000000 00:00 0
7f4be26ea000-7f4be28aa000 r-xp 00000000 08:41 8133143 /lib/x86_64-linux-gnu/libc-2.23.so
7f4be28aa000-7f4be2aaa000 ---p 001c0000 08:41 8133143 /lib/x86_64-linux-gnu/libc-2.23.so
7f4be2aaa000-7f4be2aae000 r--p 001c0000 08:41 8133143 /lib/x86_64-linux-gnu/libc-2.23.so
7f4be2aae000-7f4be2ab0000 rw-p 001c4000 08:41 8133143 /lib/x86_64-linux-gnu/libc-2.23.so
7f4be2ab0000-7f4be2ab4000 rw-p 00000000 00:00 0
7f4be2ab4000-7f4be2acd000 r-xp 00000000 08:41 8126974 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f4be2acd000-7f4be2ccc000 ---p 00019000 08:41 8126974 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f4be2ccc000-7f4be2ccd000 r--p 00018000 08:41 8126974 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f4be2ccd000-7f4be2cce000 rw-p 00019000 08:41 8126974 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f4be2cce000-7f4be2ce6000 r-xp 00000000 08:41 8133127 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f4be2ce6000-7f4be2ee5000 ---p 00018000 08:41 8133127 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f4be2ee5000-7f4be2ee6000 r--p 00017000 08:41 8133127 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f4be2ee6000-7f4be2ee7000 rw-p 00018000 08:41 8133127 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f4be2ee7000-7f4be2eeb000 rw-p 00000000 00:00 0
7f4be2eeb000-7f4be2ef2000 r-xp 00000000 08:41 8130839 /lib/x86_64-linux-gnu/librt-2.23.so
7f4be2ef2000-7f4be30f1000 ---p 00007000 08:41 8130839 /lib/x86_64-linux-gnu/librt-2.23.so
7f4be30f1000-7f4be30f2000 r--p 00006000 08:41 8130839 /lib/x86_64-linux-gnu/librt-2.23.so
7f4be30f2000-7f4be30f3000 rw-p 00007000 08:41 8130839 /lib/x86_64-linux-gnu/librt-2.23.so
7f4be30f3000-7f4be31fb000 r-xp 00000000 08:41 8133146 /lib/x86_64-linux-gnu/libm-2.23.so
7f4be31fb000-7f4be33fa000 ---p 00108000 08:41 8133146 /lib/x86_64-linux-gnu/libm-2.23.so
7f4be33fa000-7f4be33fb000 r--p 00107000 08:41 8133146 /lib/x86_64-linux-gnu/libm-2.23.so
7f4be33fb000-7f4be33fc000 rw-p 00108000 08:41 8133146 /lib/x86_64-linux-gnu/libm-2.23.so
7f4be33fc000-7f4be3422000 r-xp 00000000 08:41 8133125 /lib/x86_64-linux-gnu/ld-2.23.so
7f4be344f000-7f4be3473000 rw-p 00000000 00:00 0
7f4be34a4000-7f4be34d5000 rw-p 00000000 00:00 0
7f4be35d6000-7f4be35fc000 rw-p 00000000 00:00 0
7f4be35ff000-7f4be3621000 rw-p 00000000 00:00 0
7f4be3621000-7f4be3622000 r--p 00025000 08:41 8133125 /lib/x86_64-linux-gnu/ld-2.23.so
7f4be3622000-7f4be3623000 rw-p 00026000 08:41 8133125 /lib/x86_64-linux-gnu/ld-2.23.so
7f4be3623000-7f4be3624000 rw-p 00000000 00:00 0
7ffce54ce000-7ffce54f0000 rw-p 00000000 00:00 0 [stack]
7ffce55d1000-7ffce55d3000 r--p 00000000 00:00 0 [vvar]
7ffce55d3000-7ffce55d5000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
** Error in /home/yu/app/CSA2.6/INSTALL/../bin/wtdbg2.2/wtdbg-cns': free(): corrupted unsorted chunks: 0x00000000025beda0 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fab1f8af7e5] /lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7fab1f8b837a] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fab1f8bc53c] /home/yu/app/CSA2.6/INSTALL/../bin/wtdbg2.2/wtdbg-cns[0x404415] /home/yu/app/CSA2.6/INSTALL/../bin/wtdbg2.2/wtdbg-cns[0x4036e1] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fab1fe236ba] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fab1f93f41d] ======= Memory map: ======== 00400000-00421000 r-xp 00000000 08:41 2139283 /home/yu/app/CSA2.6/bin/wtdbg2.2/wtdbg-cns 00620000-00621000 r--p 00020000 08:41 2139283 /home/yu/app/CSA2.6/bin/wtdbg2.2/wtdbg-cns 00621000-00622000 rw-p 00021000 08:41 2139283 /home/yu/app/CSA2.6/bin/wtdbg2.2/wtdbg-cns 025be000-0268d000 rw-p 00000000 00:00 0 [heap] 7fab08000000-7fab08023000 rw-p 00000000 00:00 0 7fab08023000-7fab0c000000 ---p 00000000 00:00 0 7fab0f4fd000-7fab0f6fe000 rw-p 00000000 00:00 0 7fab0f7ff000-7fab0f800000 ---p 00000000 00:00 0 7fab0f800000-7fab10000000 rw-p 00000000 00:00 0 7fab10000000-7fab10023000 rw-p 00000000 00:00 0 7fab10023000-7fab14000000 ---p 00000000 00:00 0 7fab14000000-7fab14023000 rw-p 00000000 00:00 0 7fab14023000-7fab18000000 ---p 00000000 00:00 0 7fab18000000-7fab18021000 rw-p 00000000 00:00 0 7fab18021000-7fab1c000000 ---p 00000000 00:00 0 7fab1c029000-7fab1c0aa000 rw-p 00000000 00:00 0 7fab1c0e4000-7fab1c12b000 rw-p 00000000 00:00 0 7fab1c12b000-7fab1c12c000 ---p 00000000 00:00 0 7fab1c12c000-7fab1c92c000 rw-p 00000000 00:00 0 7fab1c92c000-7fab1c92d000 ---p 00000000 00:00 0 7fab1c92d000-7fab1d12d000 rw-p 00000000 00:00 0 7fab1d12d000-7fab1d12e000 ---p 00000000 00:00 0 7fab1d12e000-7fab1d92e000 rw-p 00000000 00:00 0 7fab1d92e000-7fab1d92f000 ---p 00000000 00:00 0 7fab1d92f000-7fab1e12f000 rw-p 00000000 00:00 0 7fab1e12f000-7fab1e130000 ---p 00000000 00:00 0 7fab1e130000-7fab1e930000 rw-p 00000000 00:00 0 7fab1ee1f000-7fab1ee36000 r-xp 00000000 08:41 8127216 /lib/x86_64-linux-gnu/libgcc_s.so.1 7fab1ee36000-7fab1f035000 ---p 00017000 08:41 8127216 /lib/x86_64-linux-gnu/libgcc_s.so.1 7fab1f035000-7fab1f036000 r--p 00016000 08:41 8127216 /lib/x86_64-linux-gnu/libgcc_s.so.1 7fab1f036000-7fab1f037000 rw-p 00017000 08:41 8127216 /lib/x86_64-linux-gnu/libgcc_s.so.1 7fab1f037000-7fab1f038000 ---p 00000000 00:00 0 7fab1f038000-7fab1f838000 rw-p 00000000 00:00 0 7fab1f838000-7fab1f9f8000 r-xp 00000000 08:41 8133143 /lib/x86_64-linux-gnu/libc-2.23.so 7fab1f9f8000-7fab1fbf8000 ---p 001c0000 08:41 8133143 /lib/x86_64-linux-gnu/libc-2.23.so 7fab1fbf8000-7fab1fbfc000 r--p 001c0000 08:41 8133143 /lib/x86_64-linux-gnu/libc-2.23.so 7fab1fbfc000-7fab1fbfe000 rw-p 001c4000 08:41 8133143 /lib/x86_64-linux-gnu/libc-2.23.so 7fab1fbfe000-7fab1fc02000 rw-p 00000000 00:00 0 7fab1fc02000-7fab1fc1b000 r-xp 00000000 08:41 8126974 /lib/x86_64-linux-gnu/libz.so.1.2.8 7fab1fc1b000-7fab1fe1a000 ---p 00019000 08:41 8126974 /lib/x86_64-linux-gnu/libz.so.1.2.8 7fab1fe1a000-7fab1fe1b000 r--p 00018000 08:41 8126974 /lib/x86_64-linux-gnu/libz.so.1.2.8 7fab1fe1b000-7fab1fe1c000 rw-p 00019000 08:41 8126974 /lib/x86_64-linux-gnu/libz.so.1.2.8 7fab1fe1c000-7fab1fe34000 r-xp 00000000 08:41 8133127 /lib/x86_64-linux-gnu/libpthread-2.23.so 7fab1fe34000-7fab20033000 ---p 00018000 08:41 8133127 /lib/x86_64-linux-gnu/libpthread-2.23.so 7fab20033000-7fab20034000 r--p 00017000 08:41 8133127 /lib/x86_64-linux-gnu/libpthread-2.23.so 7fab20034000-7fab20035000 rw-p 00018000 08:41 8133127 /lib/x86_64-linux-gnu/libpthread-2.23.so 7fab20035000-7fab20039000 rw-p 00000000 00:00 0 7fab20039000-7fab20040000 r-xp 00000000 08:41 8130839 /lib/x86_64-linux-gnu/librt-2.23.so 7fab20040000-7fab2023f000 ---p 00007000 08:41 8130839 /lib/x86_64-linux-gnu/librt-2.23.so 7fab2023f000-7fab20240000 r--p 00006000 08:41 8130839 /lib/x86_64-linux-gnu/librt-2.23.so 7fab20240000-7fab20241000 rw-p 00007000 08:41 8130839 /lib/x86_64-linux-gnu/librt-2.23.so 7fab20241000-7fab20349000 r-xp 00000000 08:41 8133146 /lib/x86_64-linux-gnu/libm-2.23.so 7fab20349000-7fab20548000 ---p 00108000 08:41 8133146 /lib/x86_64-linux-gnu/libm-2.23.so 7fab20548000-7fab20549000 r--p 00107000 08:41 8133146 /lib/x86_64-linux-gnu/libm-2.23.so 7fab20549000-7fab2054a000 rw-p 00108000 08:41 8133146 /lib/x86_64-linux-gnu/libm-2.23.so 7fab2054a000-7fab20570000 r-xp 00000000 08:41 8133125 /lib/x86_64-linux-gnu/ld-2.23.so 7fab2059d000-7fab205c1000 rw-p 00000000 00:00 0 7fab205c1000-7fab20602000 rw-p 00000000 00:00 0 7fab20724000-7fab2074a000 rw-p 00000000 00:00 0 7fab2074d000-7fab2076f000 rw-p 00000000 00:00 0 7fab2076f000-7fab20770000 r--p 00025000 08:41 8133125 /lib/x86_64-linux-gnu/ld-2.23.so 7fab20770000-7fab20771000 rw-p 00026000 08:41 8133125 /lib/x86_64-linux-gnu/ld-2.23.so 7fab20771000-7fab20772000 rw-p 00000000 00:00 0 7ffc869d6000-7ffc869f8000 rw-p 00000000 00:00 0 [stack] 7ffc869fc000-7ffc869fe000 r--p 00000000 00:00 0 [vvar] 7ffc869fe000-7ffc86a00000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
It is frustrated that I can't find the server with higher RAM.
At least the first two steps look promising. The steps in 03 and 04 typically need less memory.
Try this:
remove "set -e" remove "set -o pipefail" from STEP_03.bash
and rerun this script in directory 03_... (nohup bash STEP_03.bash) , this should run all commands, even if errors occur.
If that works you can go back to your CSA-Pipeline script (cd ../..; EDIT script/CAS.sh) and run only the last steps regarding STEP4 (include the export commands at the beginning of the script; cd xxx, mkdir xxx, STEP_04.bash).
It would also be interesting what Linux distribution your server runs.
I just found a bug that occurs if the list of reassembly jobs is very long (>13253 on our system, depends on xargs).
Could you check the number of lines of this file *wtdbg-BATCH-ALL-RUNS.sh in folder 03 ? Could you also check if "total" occurs in that file? If yes, the crash in STEP03 is related to this.
I have updated the code, please reinstall CSA from github and try again.
Thank you for your prompt reply.
The linux distribution:
Linux version 4.4.0-184-generic (buildd@lgw01-amd64-060) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.12) ) #214-Ubuntu SMP Thu Jun 4 10:14:11 UTC 2020
But there is still error when I run the step 4, there is all the output message of step4:
`
Sun Jun 21 04:32:26 CST 2020
RUN WHOLE GENOME ALIGNMENT BY LAST
Sun Jun 21 06:32:12 CST 2020
RUN RAGOUT TO ORDER CONTIGS ACCORDING TO REFERENCE
There is the content of ragout.log:
[06:32:12] INFO: Cooking Ragout...
[06:32:12] INFO: Reading FASTA with contigs
[06:33:44] INFO: Converting MAF to synteny
Usage: maf2synteny [-o out_dir] [-s simpl_params] [-m block_sizes] maf_file
positional arguments:
-------maf_file>-------path to maf file
optional arguments:
--------o out_dir>-----path to the output directory [default = .] --------s simpl_params>path to a file with custom simplification parameters [default = not set] --------b block_sizes>-comma-separated list of synteny block scales [default = 5000] [06:33:44] ERROR: Some error inside native ragout-maf2synteny module [06:33:44] ERROR: Something went wrong with maf2synteny [06:33:44] ERROR: There were problems with synteny backend, exiting. `
The number of lines of N33.step2.fa.wtdbg-BATCH-ALL-RUNS.sh is 7223. The step3 succeeded when I deleted the two line code according to what you said.
The error in RAGOUT is due to the environment variables not set. Before running STEP_04.bash, you will need to set:
export PATH="your path"/CSA2.6/INSTALL/../bin/RAGOUT_V1.0/lib:$PATH export PYTHONPATH="your path"/CSA2.6/INSTALL/../bin/RAGOUT_V1.0/lib
You have to replace "your path", by the path where you installed CSA.
Could you please post the Contig statistics after step_03?
STEP3 GAP CLOSED CONTIG STATS: Total: 2443778488 Count: 3944 Average: 619619.29 Median: 53101 N00: 28648872 1 N10: 10834336 17 N20: 8016401 42 N30: 6651974 75 N40: 5104462 118 N50: 4126746 170 N60: 3095737 239 N70: 2227178 331 N80: 1329461 473 N90: 521975 751 N100: 5002 3944
Looks good, the gap reassembly worked and improved Contig N50 3.2-fold! I guess your using a relatively low coverage dataset 10-20X and this is why you can run the assembly on only 188G RAM.
Thank you for your help! CSA succeeded!
Great! Thanks for your feedback, it helped to improve the pipeline. Would you share the final contig and scaffold statistics?
FINAL CONTIG STATS: Total: 2443172620 Count: 3745 Average: 652382.54 Median: 51018 N00: 28648872 1 N10: 12126993 16 N20: 9488405 38 N30: 7049521 69 N40: 5656134 108 N50: 4723955 155 N60: 3579919 215 N70: 2599312 294 N80: 1489703 417 N90: 590722 664 N100: 5002 3745
FINAL SCAFFOLD STATS: Total: 2492961929 Count: 1339 Average: 1861808.76 Median: 15090 N00: 187901165 1 N10: 173405562 2 N20: 165873924 3 N30: 145636299 5 N40: 140845982 7 N50: 124829477 9 N60: 118960978 11 N70: 111433577 13 N80: 106029421 15 N90: 88410764 18 N100: 5002 1339
I assemble mouse genome with nanopore long reads, but crash every time and I cannot figure out the reason. There is the error message:
`Fri Jun 19 09:20:34 CST 2020 MAP RAW READS TO ASSEMBLY
Fri Jun 19 09:53:52 CST 2020 WRITE GAP/CTGEND MATCHING RAW READS TO FILES
Fri Jun 19 09:58:39 CST 2020 RUN LOCAL ASSEMBLIES
Error in `/home/yu/app/CSA2.6/INSTALL/../bin/wtdbg2.2/wtdbg-cns': free(): corrupted unsorted chunks: 0x00000000017deda0 Error in `/home/yu/app/CSA2.6/INSTALL/../bin/wtdbg2.2/wtdbg-cns': free(): corrupted unsorted chunks: 0x0000000001d91da0 ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f50427fe7e5] /lib/x86_64-linux-gnu/libc.so.6/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f504280737a] /lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7f188afa737a] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f5042d726ba] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f188afab53c] /home/yu/app/CSA2.6/INSTALL/../bin/wtdbg2.2/wtdbg-cns[0x404415] /home/yu/app/CSA2.6/INSTALL/../bin/wtdbg2.2/wtdbg-cns[0x4036e1] /lib/x86_64-linux-gnu/libc.so.6(clone/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f188b5126ba] 00400000-00421000 r-xp 00000000 08:41 2139283 /home/yu/app/CSA2.6/bin/wtdbg2.2/wtdbg-cns 00620000-00621000 r--p 00020000 08:41 2139283 /home/yu/app/CSA2.6/bin/wtdbg2.2/wtdbg-cns 00621000-00622000 rw-p 00021000 08:41 2139283 /home/yu/app/CSA2.6/bin/wtdbg2.2/wtdbg-cns 017de000-01889000 rw-p 00000000 00:00 0 [heap] 7f502c000000-7f502c023000 rw-p 00000000 00:00 0` Maybe my RAM is too low(188G)? If it is the reason, I think you should mention the minimal RAW requirement. Thank you!