search

marbl / metAMOS

A metagenomic and isolate assembly and analysis pipeline built with AMOS
http://marbl.github.io/metAMOS
Other
93 stars 45 forks source link

Scaffold failed with return code -11 #248

Open KDSR opened 8 years ago

KDSR commented 8 years ago

I have tried running scaffold a few times on a sample, and I keep getting an error. The details are below.

During scaffold, the following command failed with return code -11:

/opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/OrientContigs -minRedundancy 5 -all -redundancy 10 -b /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk -repeats /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.reps

****DETAILS****** Last 10 commands run before the error (/master/kreeves/metAMOS/20160812_microbiome_test_c/Logs/COMMANDS.log) |2016-08-15 10:23:19|# [SCAFFOLD] |2016-08-15 10:23:19| rm -rf /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk |2016-08-15 10:48:26| /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/toAmos_new -Q /master/kreeves/metAMOS/20160812_microbiome_test_c/Preprocess/out/lib1.seq -i --min 200 --max 800 --libname lib1 -b /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk |2016-08-15 10:55:30| /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/toAmos_new -c /master/kreeves/metAMOS/20160812_microbiome_test_c/Assemble/out/proba.asm.tigr -b /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk |2016-08-15 11:10:05| /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/asmQC -b /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk -scaff -recompute -update -numsd 2 |2016-08-15 11:10:05| perl /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/bank-unlock /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk |2016-08-15 11:18:35| /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/clk -b /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk |2016-08-15 11:23:23| /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/Bundler -b /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk |2016-08-15 21:18:15| /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/MarkRepeats -redundancy 50 -b /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk > /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.reps |2016-08-16 00:21:22| /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/OrientContigs -minRedundancy 5 -all -redundancy 10 -b /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.bnk -repeats /master/kreeves/metAMOS/20160812_microbiome_test_c/Scaffold/in/proba.reps

Last 10 lines of output (/master/kreeves/metAMOS/20160812_microbiome_test_c/Logs/SCAFFOLD.log) FOR SKIPPED EDGE 1061300 SET EDGE STATUS TO BE 5 FOR SKIPPED EDGE 1108067 SET EDGE STATUS TO BE 6 FOR SKIPPED EDGE 1222376 SET EDGE STATUS TO BE 2 FOR SKIPPED EDGE 1237593 SET EDGE STATUS TO BE 5 FOR SKIPPED EDGE 1272663 SET EDGE STATUS TO BE 5 FOR SKIPPED EDGE 1348062 SET EDGE STATUS TO BE 5 FOR SKIPPED EDGE 1364684 SET EDGE STATUS TO BE 5 FOR SKIPPED EDGE 1461303 SET EDGE STATUS TO BE 5 FOR SKIPPED EDGE 1471533 SET EDGE STATUS TO BE 6 FOR SKIPPED EDGE 1500178 SET EDGE STATUS TO BE 5

Please veryify input data and restart MetAMOS. If the problem persists please contact the MetAMOS development team.

Any help or guidance will be appreciated! Thanks, Kim

KDSR commented 8 years ago

I ran the MetAMOS pipeline again, trying to skip the scaffold step, but I got another error during the postprocess step. The details are below, and I would appreciate any help or feedback to get the pipeline working. Kim

Warning: untrusted X11 forwarding setup failed: xauth key data not generated Last login: Mon Aug 15 10:22:33 2016 from mac206.txbiomedgenetics.org Rocks 6.2 (SideWinder) Profile built 20:54 25-Jan-2016

Kickstarted 16:22 25-Jan-2016 stevens.txbiomedgenetics.org% cd metAMOS /master/kreeves/metAMOS stevens.txbiomedgenetics.org% runPipeline -n Scaffold -d 20160812_microbiome_test_c

Starting metAMOS pipeline Warning: BLASR is not found, some functionality will not be available Warning: Newbler is not found, some functionality will not be available Warning: MetaGeneMark is not found, some functionality will not be available Warning: SignalP+ is not found, some functionality will not be available Warning: PHmmer is not found, some functionality will not be available Warning: REAPR is not found, some functionality will not be available Warning: FreeBayes is not found, some functionality will not be available Warning: MPI is not available, some functionality may not be available [Available RAM: 130 GB] ok [Available CPUs: 48] ok

Tasks which will be run:

Task = assemble.SplitAssemblers Task = assemble.Assemble Task = assemble.CheckAsmResults Task = assemble.SplitMappers Task = mapreads.MapReads Task = mapreads.CheckMapResults Task = mapreads.SplitForORFs Task = findorfs.FindORFS Task = validate.Validate Task = findreps.FindRepeats Task = annotate.Annotate Task = fannotate.FunctionalAnnotation Task = scaffold.Scaffold Task = findscforfs.FindScaffoldORFS Task = abundance.Abundance Task = propagate.Propagate Task = classify.Classify Task = postprocess.Postprocess

metAMOS configuration summary: metAMOS Version: v1.5rc3 "Praline Brownie" workflows: core,optional,imetamos Time and Date: 2016-09-02 Working directory: /master/kreeves/metAMOS/20160812_microbiome_test_c Prefix: proba K-Mer: 31 Threads: 47 Taxonomic level: class Verbose: False Steps to skip: MultiAlign, FindRepeats, Scaffold, FunctionalAnnotation, Propagate, FindScaffoldORFS Steps to force:

[citation] MetAMOS Treangen, TJ ⇔ Koren, S, Sommer, DD, Liu, B, Astrovskaya, I, Ondov, B, Darling AE, Phillippy AM, Pop, M. MetAMOS: a modular and open source metagenomic assembly and analysis pipeline. Genome biology, 14(1), R2, 2013.

Step-specific configuration: [abundance] MetaPhyler /opt/metAMOS/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64 Liu B, Gibbons T, Ghodsi M, Treangen T, Pop M. Accurate and fast estimation of taxonomic profiles from metagenomic shotgun sequences. BMC Genomics. 2011;12 Suppl 2:S4. Epub 2011 Jul 27.

[multialign] M-GCAT /opt/metAMOS/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64 Treangen TJ, Messeguer X. M-GCAT: interactively and efficiently constructing large-scale multiple genome comparison frameworks in closely related species. BMC Bioinformatics, 2006.

[fannotate] BLAST /opt/metAMOS/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64 Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ. Basic local alignment search tool. J Mol Biol. 1990 Oct 5;215(3):403-10.

[scaffold] Bambus 2 /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin Koren S, Treangen TJ, Pop M. Bambus 2: scaffolding metagenomes. Bioinformatics 27(21): 2964-2971 2011.

[findorfs] FragGeneScan /opt/metAMOS/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64 Rho M, Tang H, Ye Y: FragGeneScan: predicting genes in short and error-prone reads. Nucleic Acids Research 2010, 38:e191-e191.

[annotate] Kraken /opt/metAMOS/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64/kraken/bin Wood DE, Salzberg SL: Kraken: ultrafast metagenomic sequence classification using exact alignments. Genome Biology 2014, 15:R46.

[assemble] SOAPdenovo /opt/metAMOS/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64 Li Y, Hu Y, Bolund L, Wang J: State of the art de novo assembly of human genomes from massively parallel sequencing data.Human genomics 2010, 4:271-277.

[mapreads] Bowtie /opt/metAMOS/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64 Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efficient alignment of short DNA sequences to the human genome. Genome Biol. 2009;10(3):R25. Epub 2009 Mar 4.

[preprocess] metAMOS built-in filtering N/A

[validate] LAP /opt/metAMOS/metAMOS-1.5rc3/LAP Ghodsi M, Hill CM, Astrovskaya I, Lin H, Sommer DD, Koren S, Pop M. De novo likelihood-based measures for comparing genome assemblies. BMC research notes 6:334, 2013.

[other] Krona /opt/metAMOS/metAMOS-1.5rc3/KronaTools/bin Ondov BD, Bergman NH, Phillippy AM. Interactive metagenomic visualization in a Web browser. BMC Bioinformatics. 2011 Sep 30;12:385.

Job  = [preprocess.success -> *.run] completed

Completed Task = assemble.SplitAssemblers Uptodate Task = assemble.Assemble Uptodate Task = assemble.CheckAsmResults Uptodate Task = assemble.SplitMappers Uptodate Task = mapreads.MapReads Uptodate Task = mapreads.CheckMapResults Uptodate Task = mapreads.SplitForORFs Uptodate Task = findorfs.FindORFS Uptodate Task = validate.Validate Starting Task = findrepeats.FINDREPEATS Job = [proba.fna -> proba.repeats] completed Completed Task = findreps.FindRepeats Uptodate Task = annotate.Annotate Starting Task = functionalannotation.FUNCTIONALANNOTATION Job = [proba.faa -> [blast.out, krona.ec.input]] completed Completed Task = fannotate.FunctionalAnnotation Starting Task = scaffold.SCAFFOLD Job = [[proba.asm.contig] -> scaffold.ok] completed Completed Task = scaffold.Scaffold

Starting Task = findscaffoldorfs.FINDSCAFFOLDORFS Job = [proba.linearize.scaffolds.final -> proba.fna] completed Completed Task = findscforfs.FindScaffoldORFS Starting Task = abundance.ABUNDANCE Job = [proba.asm.contig -> proba.taxprof.pct.txt] completed Completed Task = abundance.Abundance Starting Task = propagate.PROPAGATE Job = [proba.annots -> propagate.ok] completed Completed Task = propagate.Propagate Starting Task = classify.CLASSIFY Job = [proba.clusters -> classify.ok] completed Completed Task = classify.Classify Starting Task = postprocess.POSTPROCESS

****ERROR****** During postprocess, the following command failed with return code 2:

perl -I /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/lib /opt/metAMOS/metAMOS-1.5rc3/Utilities/perl/statistics.pl /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/proba.scf.fa > /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/asmstats.out

****DETAILS****** Last 10 commands run before the error (/master/kreeves/metAMOS/20160812_microbiome_test_c/Logs/COMMANDS.log) |2016-09-02 16:36:01| ln /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/abundance.krona.html /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/Abundance.html |2016-09-02 16:36:01| touch /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/ref.name |2016-09-02 16:36:01| mv /master/kreeves/metAMOS/20160812_microbiome_test_c/Preprocess/out/*.fastqc /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html |2016-09-02 16:36:01| unlink /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/propagate.in.clusters |2016-09-02 16:36:01| ln /master/kreeves/metAMOS/20160812_microbiome_test_c/Propagate/in/proba.clusters /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/propagate.in.clusters |2016-09-02 16:36:01| unlink /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/propagate.out.clusters |2016-09-02 16:36:01| ln /master/kreeves/metAMOS/20160812_microbiome_test_c/Propagate/out/proba.clusters /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/propagate.out.clusters |2016-09-02 16:36:01| unlink /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/FunctionalAnnotation.html |2016-09-02 16:36:01| ln /master/kreeves/metAMOS/20160812_microbiome_test_c/FunctionalAnnotation/out/ec.krona.html /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/FunctionalAnnotation.html |2016-09-02 16:36:01| perl -I /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/lib /opt/metAMOS/metAMOS-1.5rc3/Utilities/perl/statistics.pl /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/proba.scf.fa > /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/asmstats.out

Last 10 lines of output (/master/kreeves/metAMOS/20160812_microbiome_test_c/Logs/POSTPROCESS.log) unlink: cannot unlink /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/class.classified': No such file or directory unlink: cannot unlink/master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/Annotate.html': No such file or directory unlink: cannot unlink /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/Abundance.html': No such file or directory mv: cannot stat/master/kreeves/metAMOS/20160812_microbiome_test_c/Preprocess/out/*.fastqc': No such file or directory unlink: cannot unlink /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/propagate.in.clusters': No such file or directory unlink: cannot unlink/master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/propagate.out.clusters': No such file or directory unlink: cannot unlink /master/kreeves/metAMOS/20160812_microbiome_test_c/Postprocess/out/html/FunctionalAnnotation.html': No such file or directory ln: accessing/master/kreeves/metAMOS/20160812_microbiome_test_c/FunctionalAnnotation/out/ec.krona.html': No such file or directory Can't locate Statistics/Descriptive.pm in @INC (@INC contains: /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/lib /opt/metAMOS/metAMOS-1.5rc3/src/phylosift/lib/ /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /opt/metAMOS/metAMOS-1.5rc3/Utilities/perl/statistics.pl line 7. BEGIN failed--compilation aborted at /opt/metAMOS/metAMOS-1.5rc3/Utilities/perl/statistics.pl line 7.

Please veryify input data and restart MetAMOS. If the problem persists please contact the MetAMOS development team. ****ERROR******

rm: cannot remove `/master/kreeves/metAMOS/20160812_microbiome_test_c/Logs/postprocess.ok': No such file or directory Oops, MetAMOS finished with errors! see text in red above for details.

skoren commented 8 years ago

You are missing the Statistics/Descriptive.pm perl library. It should be installed by metAMOS but it looks like it wasn't in your case. If you download/install it it should fix the error.

KDSR commented 8 years ago

Hi Sergey,

Thanks so much for getting back to me! It looks like the Statistics/Descriptive library is already installed (see below). Is there a special perl that needs to be running, or is this a special perl belonging to metAMOS that needs to be in the path? How do you install the library using metAMOS?

Thanks again!! Kim

[root@stevens ~]# cpanm Statistics::Descriptive Statistics::Descriptive is up to date. (3.0612) [root@stevens ~]#

Here is a list of the perl modules installed on Stevens…

[root@stevens ~]# instmodsh Available commands are: l - List all installed modules m - Select a module q - Quit the program cmd? l Installed modules are: Algorithm::Diff App::cpanminus Archive::Tar Archive::Zip B::Hooks::EndOfScope Bio Bio::FASTASequence Bio::FASTASequence::File Bio::Phylo Bio::Run CPAN Capture::Tiny Class::Accessor Class::Data::Inheritable Compress::Raw::Bzip2 Compress::Raw::Zlib Convert::Binary::C Cwd Data::Dumper Devel::StackTrace Digest::SHA Exception::Class Exporter::Tiny ExtUtils::CBuilder ExtUtils::MakeMaker File::Copy::Link File::HomeDir File::Sort File::Temp File::Which HTML-TableExtract IO::Compress IPC::System::Simple Module::Build Module::Implementation Module::Runtime Net Package::Stash Parse::CPAN::Meta Perl SUPER Socket Sort::Naturally Statistics::Descriptive Sub::Exporter::Progressive Sub::Identify Term::ReadKey Term::ReadLine Test::Differences Test::Exception Test::Harness Test::MockModule Test::Most Test::Simple Text::Diff Time::Piece Variable::Magic XML::SAX::Base XML::SAX::Expat XML::Simple namespace::clean

On Sep 22, 2016, at 2:44 PM, Sergey Koren notifications@github.com<mailto:notifications@github.com> wrote:

You are missing the Statistics/Descriptive.pm perl library. It should be installed by metAMOS but it looks like it wasn't in your case. If you download/install it it should fix the error.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/marbl/metAMOS/issues/248#issuecomment-249007780, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AUr5Vf4Md7BYzneIl_Ti2GFqmCHDIzSlks5qstqWgaJpZM4Jy6Wa.

CONFIDENTIALITY NOTICE: This e-mail and any files and/or attachments transmitted, may contain privileged and confidential information and is intended solely for the exclusive use of the individual or entity to whom it is addressed. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or copying of this e-mail and/or attachments is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender stating that this transmission was misdirected; return the e-mail to sender; destroy all paper copies and delete all electronic copies from your system without disclosing its contents.

skoren commented 8 years ago

The paths perl is checking is listed in the error message:

Can't locate Statistics/Descriptive.pm in @INC (@INC contains: /opt/metAMOS/metAMOS-1.5rc3/AMOS/Linux-x86_64/lib /opt/metAMOS/metAMOS-1.5rc3/src/phylosift/lib/ /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .)

It looks like it includes your system perl5 libraries, is Statistics::Descriptive available in the paths listed above? If not, you'd need to add it to your PERL5LIB path.

INSTALL.py installs it along with AMOS so if you remove the AMOS folder in metAMOS and re-run INSTALL.py it will check for Statistics::Descriptive and install if it is not found, assuming your environment where you install metAMOS is the same as where you run.

shizhaojingszj commented 7 years ago

Exactly the same error, using HMP sample: SRS077730, two files: fq1=SRS077730.denovo_duplicates_marked.trimmed.1.fastq fq2=SRS077730.denovo_duplicates_marked.trimmed.2.fastq

First time run: $Script/initPipeline -q -1 $fq1 -2 $fq2 -d $outdir -i 160:260 $Script/runPipeline -a soap -c fcp -g fraggenescan -p 15 -d $outdir -k 55 -f Assemble,MapReads,FindORFS,Annotate,FunctionalAnnotation,Propagate,Classify,Abundance,FindScaffoldORFS

failed at Mapreads without any error:

Second Time Run: $Script/runPipeline -a soap -c fcp -g fraggenescan -p 15 -d $outdir -k 55 -f FindORFS,Annotate,FunctionalAnnotation,Propagate,Classify,Abundance,FindScaffoldORFS

failed at Scaffold step, error message is quite the same, error code -11

dmesg result: [1923489.287583] OrientContigs[114077]: segfault at 10 ip 000000000041f63a sp 00007ffc350e3cd0 error 4 in OrientContigs[400000+119000]

Hope this can be fixed.