search

marbl / metAMOS

A metagenomic and isolate assembly and analysis pipeline built with AMOS
http://marbl.github.io/metAMOS
Other
93 stars 45 forks source link

Error at the scaffold step when test #207

Open yyq410 opened 9 years ago

yyq410 commented 9 years ago

Hi~, when I test the pipeline on my dataset, it failed at the scaffold step: is there something wrong with my installation ? or ? ****ERROR****** During scaffold, the following command failed with return code -11:

/opt/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/OrientContigs -minRedundancy 5 -all -redundancy 10 -b /data/yyq/test/test1/Scaffold/in/proba.bnk -repeats /data/y yq/test/test1/Scaffold/in/proba.reps

 command:
 python /opt/metAMOS-1.5rc3/initPipeline -q -1 ERR526291_1.fastq.gz -2 ERR526291_2.fastq.gz -d test1  -i 300:500
 python /opt/metAMOS-1.5rc3/runPipeline -a soap -c kraken -g fraggenescan -p 10 -d test1 -f Assemble,MapReads,FindORFS,Abundance,Classify,Annotate

 the output is below:

Warning: Newbler is not found, some functionality will not be available Project dir /data/yyq/test/test1 successfully created! Use runPipeline.py to start Pipeline

Starting Task = runpipeline.RUNPIPELINE Starting metAMOS pipeline Warning: Newbler is not found, some functionality will not be available [Available RAM: 533 GB] ok [Available CPUs: 20] ok

Tasks which will be run:

Task = preprocess.Preprocess Task = assemble.SplitAssemblers Task = assemble.Assemble Task = assemble.CheckAsmResults Task = assemble.SplitMappers Task = mapreads.MapReads Task = mapreads.CheckMapResults Task = mapreads.SplitForORFs Task = findorfs.FindORFS Task = validate.Validate Task = findreps.FindRepeats Task = annotate.Annotate Task = fannotate.FunctionalAnnotation Task = scaffold.Scaffold Task = findscforfs.FindScaffoldORFS Task = abundance.Abundance Task = propagate.Propagate Task = classify.Classify Task = postprocess.Postprocess

metAMOS configuration summary: metAMOS Version: v1.5rc3 "Praline Brownie" workflows: core,imetamos,opti onal,deprecated Time and Date: 2015-05-04 Working directory: /data/yyq/test/test1 Prefix: proba K-Mer: 31 Threads: 10 Taxonomic level: class Verbose: False Steps to skip: MultiAlign, FunctionalAnnotation, FindRepeats, FindScaff oldORFS Steps to force: Abundance, FindORFS, Annotate, MapReads, Assemble, Classify

[citation] ......

Starting Task = preprocess.PREPROCESS Job = [[ERR526291_1.fastq, ERR526291_2.fastq] -> preprocess.success] comple ted Completed Task = preprocess.Preprocess Starting Task = assemble.ASSEMBLE Job = [preprocess.success -> *.run] completed Completed Task = assemble.SplitAssemblers Job = [soapdenovo.31.run -> soapdenovo.31.asm.contig] completed Completed Task = assemble.Assemble Job = [[soapdenovo.31.asm.contig] -> [assemble.ok]] completed Completed Task = assemble.CheckAsmResults Uptodate Task = assemble.SplitMappers Starting Task = mapreads.MAPREADS Job = [soapdenovo.31.asm.contig -> soapdenovo.31.contig.cvg] completed Completed Task = mapreads.MapReads Job = [[soapdenovo.31.contig.cvg] -> [mapreads.ok]] completed Completed Task = mapreads.CheckMapResults Uptodate Task = mapreads.SplitForORFs Starting Task = findorfs.FINDORFS Job = [soapdenovo.31.contig.cvg -> soapdenovo.31.faa] completed Completed Task = findorfs.FindORFS Starting Task = validate.VALIDATE Job = [[soapdenovo.31.faa] -> [validate.ok]] completed Completed Task = validate.Validate Starting Task = findrepeats.FINDREPEATS Job = [proba.fna -> proba.repeats] completed Completed Task = findreps.FindRepeats Starting Task = annotate.ANNOTATE Job = [proba.faa -> proba.hits] completed Completed Task = annotate.Annotate Starting Task = functionalannotation.FUNCTIONALANNOTATION Job = [proba.faa -> [blast.out, krona.ec.input]] completed Completed Task = fannotate.FunctionalAnnotation Starting Task = scaffold.SCAFFOLD

****ERROR****** During scaffold, the following command failed with return code -11:

/opt/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/OrientContigs -minRedundancy 5 -all -redundancy 10 -b /data/yyq/test/test1/Scaffold/in/proba.bnk -repeats /data/y yq/test/test1/Scaffold/in/proba.reps ****DETAILS****** Last 10 commands run before the error (/data/yyq/test/test1/Logs/COMMANDS.log) |2015-05-04 14:51:08|# [SCAFFOLD] |2015-05-04 14:51:08| rm -rf /data/yyq/test/test1/Scaffold/in/proba.bnk [12/542] |2015-05-04 14:54:47| /opt/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/toAmos_new -Q /d ata/yyq/test/test1/Preprocess/out/lib1.seq -i --min 5 --max 1679 --libname lib1 -b /data/yyq/test/test1/Scaffold/in/proba.bnk |2015-05-04 14:56:17| /opt/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/toAmos_new -c /d ata/yyq/test/test1/Assemble/out/proba.asm.tigr -b /data/yyq/test/test1/Scaffold/ in/proba.bnk |2015-05-04 14:59:45| /opt/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/asmQC -b /data/y yq/test/test1/Scaffold/in/proba.bnk -scaff -recompute -update -numsd 2 |2015-05-04 14:59:45| perl /opt/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/bank-unlock /data/yyq/test/test1/Scaffold/in/proba.bnk |2015-05-04 15:01:57| /opt/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/clk -b /data/yyq /test/test1/Scaffold/in/proba.bnk |2015-05-04 15:03:03| /opt/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/Bundler -b /data /yyq/test/test1/Scaffold/in/proba.bnk |2015-05-04 21:16:41| /opt/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/MarkRepeats -red undancy 50 -b /data/yyq/test/test1/Scaffold/in/proba.bnk > /data/yyq/test/test1 /Scaffold/in/proba.reps |2015-05-04 21:52:32| /opt/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/OrientContigs -m inRedundancy 5 -all -redundancy 10 -b /data/yyq/test/test1/Scaffold/in/proba.b nk -repeats /data/yyq/test/test1/Scaffold/in/proba.reps

Last 10 lines of output (/data/yyq/test/test1/Logs/SCAFFOLD.log) FOR SKIPPED EDGE 926440 SET EDGE STATUS TO BE 5 FOR SKIPPED EDGE 944273 SET EDGE STATUS TO BE 5 FOR SKIPPED EDGE 982719 SET EDGE STATUS TO BE 5 FOR SKIPPED EDGE 983738 SET EDGE STATUS TO BE 5 FOR SKIPPED EDGE 1019566 SET EDGE STATUS TO BE 5 FOR SKIPPED EDGE 1026702 SET EDGE STATUS TO BE 5 FOR SKIPPED EDGE 1070057 SET EDGE STATUS TO BE 5 FOR SKIPPED EDGE 1072044 SET EDGE STATUS TO BE 5 FOR SKIPPED EDGE 1072647 SET EDGE STATUS TO BE 5 FOR SKIPPED EDGE 1112734 SET EDGE STATUS TO BE 5

Please veryify input data and restart MetAMOS. If the problem persists please co ntact the MetAMOS development team. ****ERROR******

rm: cannot remove ‘/data/yyq/test/test1/Logs/scaffold.ok’: No such file or direc tory Oops, MetAMOS finished with errors! see text in red above for details.

yyq410 commented 9 years ago

when I skip the scaffold with '-n Scaffold', I met another problem:

****ERROR****** During postprocess, the following command failed with return code 2:

perl -I /opt/metAMOS-1.5rc3/AMOS/Linux-x86_64/lib /opt/metAMOS-1.5rc3/Utilitie s/perl/statistics.pl /data/yyq/test/test1/Postprocess/out/proba.scf.fa > /data/yy q/test/test1/Postprocess/out/html/asmstats.out

command: python /opt/metAMOS-1.5rc3/initPipeline -q -1 ERR526291_1.fastq.gz -2 ERR526291_2.fastq.gz -d test1 -i 300:500 python /opt/metAMOS-1.5rc3/runPipeline -a soap -c kraken -g fraggenescan -p 10 -d test1 -f Assemble,MapReads,FindORFS,Abundance,Classify,Annotate -n Scaffold

output is below: Warning: Newbler is not found, some functionality will not be available Project dir /data/yyq/test/test1 successfully created! Use runPipeline.py to start Pipeline

Starting Task = runpipeline.RUNPIPELINE Starting metAMOS pipeline Warning: Newbler is not found, some functionality will not be available [Available RAM: 533 GB] ok [Available CPUs: 20] ok

Tasks which will be run:

Task = preprocess.Preprocess Task = assemble.SplitAssemblers Task = assemble.Assemble Task = assemble.CheckAsmResults Task = assemble.SplitMappers Task = mapreads.MapReads Task = mapreads.CheckMapResults Task = mapreads.SplitForORFs Task = findorfs.FindORFS Task = validate.Validate Task = findreps.FindRepeats Task = annotate.Annotate Task = fannotate.FunctionalAnnotation Task = scaffold.Scaffold Task = findscforfs.FindScaffoldORFS Task = abundance.Abundance Task = propagate.Propagate Task = classify.Classify Task = postprocess.Postprocess

metAMOS configuration summary: metAMOS Version: v1.5rc3 "Praline Brownie" workflows: core,imetamos,optio nal,deprecated Time and Date: 2015-05-05 Working directory: /data/yyq/test/test1 Prefix: proba K-Mer: 31 Threads: 10 Taxonomic level: class Verbose: False Steps to skip: MultiAlign, FindRepeats, Scaffold, FunctionalAnnotation, Propagate, FindScaffoldORFS Steps to force: Abundance, FindORFS, Annotate, MapReads, Assemble, Classi fy

[citation] ......

Starting Task = preprocess.PREPROCESS Job = [[ERR526291_1.fastq, ERR526291_2.fastq] -> preprocess.success] complet ed Completed Task = preprocess.Preprocess Starting Task = assemble.ASSEMBLE Job = [preprocess.success -> *.run] completed Completed Task = assemble.SplitAssemblers Job = [soapdenovo.31.run -> soapdenovo.31.asm.contig] completed Completed Task = assemble.Assemble Job = [[soapdenovo.31.asm.contig] -> [assemble.ok]] completed Completed Task = assemble.CheckAsmResults Uptodate Task = assemble.SplitMappers Starting Task = mapreads.MAPREADS Job = [soapdenovo.31.asm.contig -> soapdenovo.31.contig.cvg] completed Completed Task = mapreads.MapReads Job = [[soapdenovo.31.contig.cvg] -> [mapreads.ok]] completed Completed Task = mapreads.CheckMapResults Uptodate Task = mapreads.SplitForORFs Starting Task = findorfs.FINDORFS Job = [soapdenovo.31.contig.cvg -> soapdenovo.31.faa] completed Completed Task = findorfs.FindORFS Starting Task = validate.VALIDATE Job = [[soapdenovo.31.faa] -> [validate.ok]] completed Completed Task = validate.Validate Starting Task = findrepeats.FINDREPEATS Job = [proba.fna -> proba.repeats] completed Completed Task = findreps.FindRepeats Starting Task = annotate.ANNOTATE Job = [proba.faa -> proba.hits] completed Completed Task = annotate.Annotate Starting Task = functionalannotation.FUNCTIONALANNOTATION Job = [proba.faa -> [blast.out, krona.ec.input]] completed Completed Task = fannotate.FunctionalAnnotation Starting Task = scaffold.SCAFFOLD Job = [[proba.asm.contig] -> scaffold.ok] completed Completed Task = scaffold.Scaffold

Starting Task = findscaffoldorfs.FINDSCAFFOLDORFS Job = [proba.linearize.scaffolds.final -> proba.fna] completed Completed Task = findscforfs.FindScaffoldORFS Starting Task = abundance.ABUNDANCE Job = [proba.asm.contig -> proba.taxprof.pct.txt] completed Completed Task = abundance.Abundance Starting Task = propagate.PROPAGATE Job = [proba.annots -> propagate.ok] completed Completed Task = propagate.Propagate Starting Task = classify.CLASSIFY Job = [proba.clusters -> classify.ok] completed Completed Task = classify.Classify Starting Task = postprocess.POSTPROCESS

****ERROR****** During postprocess, the following command failed with return code 2:

perl -I /opt/metAMOS-1.5rc3/AMOS/Linux-x86_64/lib /opt/metAMOS-1.5rc3/Utilitie s/perl/statistics.pl /data/yyq/test/test1/Postprocess/out/proba.scf.fa > /data/yy q/test/test1/Postprocess/out/html/asmstats.out

****DETAILS****** Last 10 commands run before the error (/data/yyq/test/test1/Logs/COMMANDS.log) |2015-05-05 03:28:57| ln /data/yyq/test/test1/Postprocess/out/abundance.krona.htm l /data/yyq/test/test1/Postprocess/out/html/Abundance.html |2015-05-05 03:28:57| touch /data/yyq/test/test1/Postprocess/out/ref.name |2015-05-05 03:28:57| mv /data/yyq/test/test1/Preprocess/out/*.fastqc /data/yyq/t est/test1/Postprocess/out/html |2015-05-05 03:28:57| unlink /data/yyq/test/test1/Postprocess/out/html/propagate. in.clusters |2015-05-05 03:28:57| ln /data/yyq/test/test1/Propagate/in/proba.clusters /data/y yq/test/test1/Postprocess/out/html/propagate.in.clusters |2015-05-05 03:28:57| unlink /data/yyq/test/test1/Postprocess/out/html/propagate. out.clusters |2015-05-05 03:28:57| ln /data/yyq/test/test1/Propagate/out/proba.clusters /data/ yyq/test/test1/Postprocess/out/html/propagate.out.clusters |2015-05-05 03:28:57| unlink /data/yyq/test/test1/Postprocess/out/html/Functional Annotation.html |2015-05-05 03:28:57| ln /data/yyq/test/test1/FunctionalAnnotation/out/ec.krona.h tml /data/yyq/test/test1/Postprocess/out/html/FunctionalAnnotation.html |2015-05-05 03:28:57| perl -I /opt/metAMOS-1.5rc3/AMOS/Linux-x86_64/lib /opt/metA MOS-1.5rc3/Utilities/perl/statistics.pl /data/yyq/test/test1/Postprocess/out/prob a.scf.fa > /data/yyq/test/test1/Postprocess/out/html/asmstats.out

Last 10 lines of output (/data/yyq/test/test1/Logs/POSTPROCESS.log) unlink: cannot unlink ‘/data/yyq/test/test1/Postprocess/out/html/class.classified ’: No such file or directory unlink: cannot unlink ‘/data/yyq/test/test1/Postprocess/out/html/Annotate.html’: No such file or directory unlink: cannot unlink ‘/data/yyq/test/test1/Postprocess/out/html/Abundance.html’: No such file or directory mv: cannot stat ‘/data/yyq/test/test1/Preprocess/out/*.fastqc’: No such file or d irectory unlink: cannot unlink ‘/data/yyq/test/test1/Postprocess/out/html/propagate.in.clu sters’: No such file or directory unlink: cannot unlink ‘/data/yyq/test/test1/Postprocess/out/html/propagate.out.cl usters’: No such file or directory unlink: cannot unlink ‘/data/yyq/test/test1/Postprocess/out/html/FunctionalAnnota tion.html’: No such file or directory ln: failed to access ‘/data/yyq/test/test1/FunctionalAnnotation/out/ec.krona.html ’: No such file or directory Can't locate Statistics/Descriptive.pm in @INC (@INC contains: /opt/metAMOS-1.5rc 3/AMOS/Linux-x86_64/lib /opt/metAMOS-1.5rc3/src/phylosift/lib/ /home/yyq/perl5/li b/perl5 /root/bin /root/bin /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/li b64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/pe rl5 .) at /opt/metAMOS-1.5rc3/Utilities/perl/statistics.pl line 7. BEGIN failed--compilation aborted at /opt/metAMOS-1.5rc3/Utilities/perl/statistic s.pl line 7.

Please veryify input data and restart MetAMOS. If the problem persists please con tact the MetAMOS development team. ****ERROR******

rm: cannot remove ‘/data/yyq/test/test1/Logs/postprocess.ok’: No such file or dir ectory Oops, MetAMOS finished with errors! see text in red above for details.

skoren commented 9 years ago

Your second error is a missing Perl library called Statistics::Descriptive. You need to install that perl package which will fix your Postprocess error (http://metamos.readthedocs.org/en/v1.5rc3/content/installation.html#prerequisites).

As for the scaffold error, it is the same error as issue #198. If you can share your data we can try to reproduce it locally.

yyq410 commented 9 years ago

hi~ I used the published liver cirrhosis dataset, the link is below: http://www.ebi.ac.uk/ena/data/view/ERP005860 I just selected the first sample to test the pipeline. @skoren

yyq410 commented 9 years ago

hi~ Does the command 'OrientContigs' contain the parameter named ' -minRedundancy' ? I cann't find it in its usage... is it the possible problem causing the failure of the Scaffold ? @skoren

╰─[$: 1] % /opt/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin/OrientContigs -h

Determine contig order and orientation

USAGE:

OrientContigs -b[ank] [-all] [-noreduce] [-agressive] [-redundancy minLinks] [-repeats fileName] [-skip] The -all option will force initialization of all contigs, including those that have no links to them, otherwise they remain uninitialized The -noreduce option will turn off search for common motifs and recursively remove them, thus simplyfing the graph The -agressive option will not mark edges that move a contig more than 3 STDEVS away as bad and will try to reconcile the positions The -redundancy option specifies the minimum number of links between two contigs before they will be scaffolded The -repeats option specifies a file containing a list of contig IIDs which are considered repeats and whose edges will be unused The -skip option will skip edges that have too low a weight relative to the weights of the other edges connecting their respective nodes.

Options summary: Bank = Redundancy = 0 InitAll = 0 Compress = 1 AgressiveScf = 0 Max Overlap = -1 A bank must be specified

skoren commented 9 years ago

The minRedundancy is a parameter to OrientContigs. The parameters passed to OrientContigs are listed in the configuration file in Utilities/config/bambus.config. The parameter isn't the cause of the error because if invalid parameters were passed it would error immediately but it began running on your dataset before exiting with an error. We will try to reproduce the error on your dataset locally.