search

marbl / metAMOS

A metagenomic and isolate assembly and analysis pipeline built with AMOS
http://marbl.github.io/metAMOS
Other
93 stars 45 forks source link

MetAMOS error "std::bad_alloc" during scaffold.SCAFFOLD #211

Open drewmicro opened 9 years ago

drewmicro commented 9 years ago

Hi MetAmos Guys, I'm using metAMOS 1.5rc3 and am getting an error at the SCAFFOLD step. Any recommendations as to what to try? The tests on install passed this step, so maybe it's just my data? Thanks, -Andrew

****ERROR****** During scaffold, the following command failed with return code 134:

/home/iak/bin/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/MarkRepeats -redundancy 50 -b /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Scaffold/in/proba.bnk > /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Scaffold/in/proba.reps

****DETAILS****** Last 10 commands run before the error (/home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Logs/COMMANDS.log) |2015-05-15 03:10:01| touch /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/FunctionalAnnotation/out/blast.out |2015-05-15 03:10:02|# [SCAFFOLD] |2015-05-15 03:10:02| rm -rf /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Scaffold/in/proba.bnk |2015-05-15 03:22:31| /home/iak/bin/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/toAmos_new -s /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Preprocess/out/lib1.seq -m /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Assemble/out/proba.lib1.mappedmates -b /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Scaffold/in/proba.bnk |2015-05-15 03:25:11| /home/iak/bin/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/toAmos_new -c /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Assemble/out/proba.asm.tigr -b /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Scaffold/in/proba.bnk |2015-05-15 03:29:38| /home/iak/bin/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/asmQC -b /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Scaffold/in/proba.bnk -scaff -recompute -update -numsd 2 |2015-05-15 03:29:38| perl /home/iak/bin/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/bank-unlock /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Scaffold/in/proba.bnk |2015-05-15 03:31:07| /home/iak/bin/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/clk -b /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Scaffold/in/proba.bnk |2015-05-15 03:31:10| /home/iak/bin/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/Bundler -b /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Scaffold/in/proba.bnk |2015-05-15 03:32:38| /home/iak/bin/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/MarkRepeats -redundancy 50 -b /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Scaffold/in/proba.bnk > /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Scaffold/in/proba.reps

Last 10 lines of output (/home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Logs/SCAFFOLD.log) new NORMALS: 0 ANTINORMALS: 0 INNIES: 1 OUTIES: 0 Writing edge link with it 98919 Doing contig pair (7201549, 7201545) NORMALS: 1 ANTINORMALS: 0 INNIES: 0 OUTIES: 0 Link 28640 (-609, 216) is just right new NORMALS: 1 ANTINORMALS: 0 INNIES: 0 OUTIES: 0 Writing edge link with it 98920 terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc /bin/bash: line 1: 14467 Aborted (core dumped) /home/iak/bin/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/MarkRepeats -redundancy 50 -b /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Scaffold/in/proba.bnk > /home/iak/Desktop/EightSites/Fastq/ERNE-filter/metAMOS/HC/Scaffold/in/proba.reps

Please veryify input data and restart MetAMOS. If the problem persists please contact the MetAMOS development team. ****ERROR******

skoren commented 9 years ago

This seems like an out of memory error, how large is your dataset/how much ram does your system have available. You can check that the library pairs were input properly and have a reasonable size estimate from read mappings to the assembly by checking the first line in: Assemble/out/*.lib1.mappedmates It should have the library min/max sizes for your dataset.

If you want to complete the run, you can set -n Scaffold as a workaround to skip the scaffolding step.

drewmicro commented 9 years ago

I could very well be an out of memory error. The input datafile is 6.3 GB and the computer has 128 GB of RAM. When I tried to run idba by itself, after 1 week it still didn't finish.

Assemble/out/*.lib1.mappedmates has somewhat reasonable lengths (27 and 993) although I did use the option for initPipeline of -i 50:350

skoren commented 9 years ago

I think there is a bug in metAMOS where it is over-estimating the insert size causing the memory usage for scaffolding to increase (since there are lots of inconsistent edges). I'll check for a fix but in the meantime you can bypass scaffolding.

bstresGIT commented 6 years ago

Is this solved? I tried metAMOS in 2018, but it keeps coming back with the same issue when scaffolding relatively small metagenome. Is there any other solution in order to use metAMOS and not having to skip the scaffold step? Otherwise there is no point in using metAMOS, if scaffold does not work. THere is plenty of other software.

skoren commented 6 years ago

The insert size estimate should be fixed but it is possible there are other bugs in Bambus 2 which are not fixed. Really, we need to update the metAMOS scaffolder.

In the meantime skipping scaffolding is your best option. I suggest looking at MetaCarvel (https://github.com/marbl/MetaCarvel) for scaffolding which is the successor to Bambus2.