search

marbl / metAMOS

A metagenomic and isolate assembly and analysis pipeline built with AMOS
http://marbl.github.io/metAMOS
Other
93 stars 45 forks source link

Oops, MetAMOS finished with errors! #130

Closed kittim closed 9 years ago

kittim commented 10 years ago

Hello,

My run of metAMOS stops with error at various points. If I try Preprocess and Postprocess only, it stops at Validate. If I try Preprocess, Assemble, Postprocess only, it stops at Mapreads. I tried Gageb and iMetAMOS workflows as well, which stop at Classify and Postprocess, resp.

Below are my four trials, with commands, end of output and Logs/COMMANDS.log. I also add the full output from iMetAMOS workflow. It shows that KmerGenie and GraphViz cannot be found. (It may be a seperate issue, but I have problem with SOAPdenovo2 and MaSuRCA as well.)

Thank you in advance.

///////////////////////////// ////// Preprocess only ////// ///////////////////////////// DIR/initPipeline -1 tmp1.fastq -2 tmp2.fastq -d outDIR -i 300:500 DIR/runPipeline -v -d outDIR -n Assemble,MapReads,FindORFS,Validate,Annotate,Classify

/// tail output /// Starting Task = mapreads.MAPREADS *\ metAMOS running command: touch /home/skim/metAMOS/READ_12/outDIR/Logs/mapreads.skip

*\ metAMOS running command: touch /home/skim/metAMOS/READ_12/outDIR/Logs/mapreads.skip

Starting Task = validate.VALIDATE *\ metAMOS running command: touch /home/skim/metAMOS/READ_12/outDIR/Logs/validate.skip

Oops, MetAMOS finished with errors! see text in red above for details.

/// tail Logs/COMMANDS.log /// |2014-04-22 08:08:08| touch /home/skim/metAMOS/READ_12/outDIR/Preprocess/out/preprocess.success |2014-04-22 08:08:09|# [ASSEMBLE] |2014-04-22 08:08:09| touch /home/skim/metAMOS/READ_12/outDIR/Logs/assemble.skip |2014-04-22 08:08:10| touch /home/skim/metAMOS/READ_12/outDIR/Logs/assemble.skip |2014-04-22 08:08:11| touch /home/skim/metAMOS/READ_12/outDIR/Logs/assemble.skip |2014-04-22 08:08:12|# [MAPREADS] |2014-04-22 08:08:12| touch /home/skim/metAMOS/READ_12/outDIR/Logs/mapreads.skip |2014-04-22 08:08:13| touch /home/skim/metAMOS/READ_12/outDIR/Logs/mapreads.skip |2014-04-22 08:08:14|# [VALIDATE] |2014-04-22 08:08:14| touch /home/skim/metAMOS/READ_12/outDIR/Logs/validate.skip

////////////////////////////////////////// ////// Preprocess and Assemble only ////// ////////////////////////////////////////// DIR/initPipeline -1 tmp1.fastq -2 tmp2.fastq -d outDIR -i 300:500 DIR/runPipeline -v -a velvet -d outDIR -n MapReads,FindORFS,Validate,Annotate,Classify

/// tail output ///

*\ metAMOS running command: java -cp /usr/local/metAMOS-1.5rc1/Utilities/java:. RemoveIUPAC /home/skim/metAMOS/READ_10/outDIR/Assemble/out/velvet.31.asm.contigWIUPAC.fa >/home/skim/metAMOS/READ_10/outDIR/Assemble/out/velvet.31.asm.contig

*\ metAMOS running command: rm /home/skim/metAMOS/READ_10/outDIR/Assemble/out/velvet.31.run

*\ metAMOS running command: touch /home/skim/metAMOS/READ_10/outDIR/Assemble/out/assemble.success

Starting Task = mapreads.MAPREADS *\ metAMOS running command: touch /home/skim/metAMOS/READ_10/outDIR/Logs/mapreads.skip

Oops, MetAMOS finished with errors! see text in red above for details.

/// tail Logs/COMMANDS.log ///

|2014-04-21 14:40:07| rm /home/skim/metAMOS/READ_10/outDIR/Assemble/out/velvet.31.afg |2014-04-21 14:40:07| ln /home/skim/metAMOS/READ_10/outDIR/Assemble/out/velvet_asm.afg /home/skim/metAMOS/READ_10/outDIR/Assemble/out/velvet.31.afg |2014-04-21 14:40:07| rm /home/skim/metAMOS/READ_10/outDIR/Assemble/out/velvet.31.asm.contig |2014-04-21 14:40:07| ln /home/skim/metAMOS/READ_10/outDIR/Assemble/out/contigs.fa /home/skim/metAMOS/READ_10/outDIR/Assemble/out/velvet.31.asm.contig |2014-04-21 14:40:08| mv /home/skim/metAMOS/READ_10/outDIR/Assemble/out/velvet.31.asm.contig /home/skim/metAMOS/READ_10/outDIR/Assemble/out/velvet.31.asm.contigWIUPAC.fa |2014-04-21 14:40:54| java -cp /usr/local/metAMOS-1.5rc1/Utilities/java:. RemoveIUPAC /home/skim/metAMOS/READ_10/outDIR/Assemble/out/velvet.31.asm.contigWIUPAC.fa >/home/skim/metAMOS/READ_10/outDIR/Assemble/out/velvet.31.asm.contig |2014-04-21 14:40:54| rm /home/skim/metAMOS/READ_10/outDIR/Assemble/out/velvet.31.run |2014-04-21 14:40:54| touch /home/skim/metAMOS/READ_10/outDIR/Assemble/out/assemble.success |2014-04-21 14:40:55|# [MAPREADS] |2014-04-21 14:40:55| touch /home/skim/metAMOS/READ_10/outDIR/Logs/mapreads.skip

/////////////////// ////// Gageb ////// /////////////////// DIR/initPipeline -d outDIR -W isolate_gageb DIR/runPipeline -d outDIR -v

/// tail output /// *\ metAMOS running command: java -cp /usr/local/metAMOS-1.5rc1/Utilities/java:. SizeFasta /home/skim/metAMOS/Gageb/outDIR/Assemble/out/lib1.unaligned.fasta |awk '{if ($NF >= 100) { print $1} }' >> /home/skim/metAMOS/Gageb/outDIR/Classify/out/proba.read.iids

*\ metAMOS running command: java -cp /usr/local/metAMOS-1.5rc1/Utilities/java:. SubFile /home/skim/metAMOS/Gageb/outDIR/Classify/out/proba.ctg.iids /home/skim/metAMOS/Gageb/outDIR/Assemble/out/proba.lib1contig.reads 1 >> /home/skim/metAMOS/Gageb/outDIR/Classify/out/proba.read.iids

*\ metAMOS running command: java -cp /usr/local/metAMOS-1.5rc1/Utilities/java:. SubFile /home/skim/metAMOS/Gageb/outDIR/Classify/out/proba.read.iids /home/skim/metAMOS/Gageb/outDIR/Propagate/out/proba.reads.clusters > /home/skim/metAMOS/Gageb/outDIR/Classify/out/proba.read.contaminant.clusters

*\ metAMOS running command: unlink /home/skim/metAMOS/Gageb/outDIR/Classify/out/contaminant.true

Oops, MetAMOS finished with errors! see text in red above for details.

/// tail Logs/COMMANDS.log /// |2014-04-20 03:21:19| /usr/local/metAMOS-1.5rc1/AMOS/Linux-x86_64/bin/dumpreads -q -e -E '/home/skim/metAMOS/Gageb/outDIR/Classify/out/Alphaproteobacteria//Alphaproteobacteria.read.eid' /home/skim/metAMOS/Gageb/outDIR/Scaffold/in/proba.bnk > '/home/skim/metAMOS/Gageb/outDIR/Classify/out/Alphaproteobacteria//Alphaproteobacteria.read.qual' |2014-04-20 03:22:07| /usr/local/metAMOS-1.5rc1/AMOS/Linux-x86_64/bin/dumpreads -f -e -E '/home/skim/metAMOS/Gageb/outDIR/Classify/out/Alphaproteobacteria//Alphaproteobacteria.read.eid' /home/skim/metAMOS/Gageb/outDIR/Scaffold/in/proba.bnk > '/home/skim/metAMOS/Gageb/outDIR/Classify/out/Alphaproteobacteria//Alphaproteobacteria.read.fastq' |2014-04-20 03:22:07| /usr/local/metAMOS-1.5rc1/AMOS/Linux-x86_64/bin/dumpreads -e -E '/home/skim/metAMOS/Gageb/outDIR/Classify/out/Alphaproteobacteria//Alphaproteobacteria.orf.eid' /home/skim/metAMOS/Gageb/outDIR/FindORFS/out/proba.fna.bnk > '/home/skim/metAMOS/Gageb/outDIR/Classify/out/Alphaproteobacteria//Alphaproteobacteria.orf.fna.fasta' |2014-04-20 03:22:08| /usr/local/metAMOS-1.5rc1/AMOS/Linux-x86_64/bin/dumpreads -e -E '/home/skim/metAMOS/Gageb/outDIR/Classify/out/Alphaproteobacteria//Alphaproteobacteria.orf.eid' /home/skim/metAMOS/Gageb/outDIR/FindORFS/out/proba.faa.bnk > '/home/skim/metAMOS/Gageb/outDIR/Classify/out/Alphaproteobacteria//Alphaproteobacteria.orf.faa.fasta' |2014-04-20 03:22:54| java -cp /usr/local/metAMOS-1.5rc1/Utilities/java:. SizeFasta /home/skim/metAMOS/Gageb/outDIR/Assemble/out/proba.asm.contig |awk '{if ($NF >= 100) { print $1} }' > /home/skim/metAMOS/Gageb/outDIR/Classify/out/proba.ctg.iids |2014-04-20 03:22:54| unlink /home/skim/metAMOS/Gageb/outDIR/Classify/out/proba.read.iids |2014-04-20 03:23:40| java -cp /usr/local/metAMOS-1.5rc1/Utilities/java:. SizeFasta /home/skim/metAMOS/Gageb/outDIR/Assemble/out/lib1.unaligned.fasta |awk '{if ($NF >= 100) { print $1} }' >> /home/skim/metAMOS/Gageb/outDIR/Classify/out/proba.read.iids |2014-04-20 03:24:49| java -cp /usr/local/metAMOS-1.5rc1/Utilities/java:. SubFile /home/skim/metAMOS/Gageb/outDIR/Classify/out/proba.ctg.iids /home/skim/metAMOS/Gageb/outDIR/Assemble/out/proba.lib1contig.reads 1 >> /home/skim/metAMOS/Gageb/outDIR/Classify/out/proba.read.iids |2014-04-20 03:26:01| java -cp /usr/local/metAMOS-1.5rc1/Utilities/java:. SubFile /home/skim/metAMOS/Gageb/outDIR/Classify/out/proba.read.iids /home/skim/metAMOS/Gageb/outDIR/Propagate/out/proba.reads.clusters > /home/skim/metAMOS/Gageb/outDIR/Classify/out/proba.read.contaminant.clusters |2014-04-20 03:26:03| unlink /home/skim/metAMOS/Gageb/outDIR/Classify/out/contaminant.true

////////////////////// ////// iMetAMOS ////// ////////////////////// DIR/initPipeline -1 tmp1.fastq -2 tmp2.fastq -d outDIR -W iMetAMOS -i 300:500 -c contigs.fasta DIR/runPipeline -d outDIR -p 8

/// tail output /// Starting Task = annotate.ANNOTATE Starting Task = functionalannotation.FUNCTIONALANNOTATION Starting Task = scaffold.SCAFFOLD

Starting Task = findscaffoldorfs.FINDSCAFFOLDORFS Starting Task = abundance.ABUNDANCE Starting Task = propagate.PROPAGATE Starting Task = classify.CLASSIFY Starting Task = postprocess.POSTPROCESS Oops, MetAMOS finished with errors! see text in red above for details.

/// tail Logs/COMMANDS.log /// |2014-04-17 15:39:22| unlink /home/skim/READ/outDIR/Postporocess/out/html/quast |2014-04-17 15:39:23| cp -r /home/skim/READ/outDIR/Validate/out/quast /home/skim/READ/outDIR/Postprocess/out/html/quast |2014-04-17 15:39:23| mv /home/skim/READ/outDIR/Preprocess/out/*.fastqc /home/skim/READ/outDIR/Postprocess/out/html |2014-04-17 15:39:23| unlink /home/skim/READ/outDIR/Postprocess/out/html/propagate.in.clusters |2014-04-17 15:39:23| ln /home/skim/READ/outDIR/Propagate/in/proba.clusters /home/skim/READ/outDIR/Postprocess/out/html/propagate.in.clusters |2014-04-17 15:39:23| unlink /home/skim/READ/outDIR/Postprocess/out/html/propagate.out.clusters |2014-04-17 15:39:23| ln /home/skim/READ/outDIR/Propagate/out/proba.clusters /home/skim/READ/outDIR/Postprocess/out/html/propagate.out.clusters |2014-04-17 15:39:23| unlink /home/skim/READ/outDIR/Postprocess/out/html/FunctionalAnnotation.html |2014-04-17 15:39:23| ln /home/skim/READ/outDIR/FunctionalAnnotation/out/ec.krona.html /home/skim/READ/outDIR/Postprocess/out/html/FunctionalAnnotation.html |2014-04-17 15:39:23| perl -I /usr/local/metAMOS-1.5rc1/AMOS/Linux-x86_64/lib /usr/local/metAMOS-1.5rc1/Utilities/perl/statistics.pl /home/skim/READ/outDIR/Postprocess/out/proba.scf.fa > /home/skim/READ/outDIR/Postprocess/out/html/asmstats.out

/////////////////////////////// /// iMetAMOS output /// //////////////////////////////// Project dir /home/skim/READ/outDIR successfully created! Use runPipeline.py to start Pipeline

Starting metAMOS pipeline Warning: Celera Assembler is not found, some functionality will not be available Warning: BLASR is not found, some functionality will not be available Warning: Newbler is not found, some functionality will not be available Warning: MetaGeneMark is not found, some functionality will not be available Warning: SignalP+ is not found, some functionality will not be available Warning: R package is not found, some functionality will not be available Warning: REAPR is not found, some functionality will not be available Warning: FRCbam is not found, some functionality will not be available [Available RAM: 264 GB]  ok [Available CPUs: 32]  ok

Tasks which will be run:

Task = preprocess.Preprocess Task = assemble.SplitAssemblers Task = assemble.Assemble Task = assemble.CheckAsmResults Task = assemble.SplitMappers Task = mapreads.MapReads Task = mapreads.CheckMapResults Task = mapreads.SplitForORFs Task = findorfs.FindORFS Task = validate.Validate Task = findreps.FindRepeats Task = annotate.Annotate Task = fannotate.FunctionalAnnotation Task = scaffold.Scaffold Task = findscforfs.FindScaffoldORFS Task = abundance.Abundance Task = propagate.Propagate Task = classify.Classify Task = postprocess.Postprocess

Warning: Graphviz is not found, some functionality will not be available metAMOS configuration summary: metAMOS Version: v1.5rc1 "Praline Brownie" workflows: core,optional,imetamos Time and Date: 2014-04-17 Working directory: /home/skim/READ/outDIR Prefix: proba K-Mer: 31 Threads: 8 Taxonomic level: class Verbose: False Steps to skip: FunctionalAnnotation, FindScaffoldORFS, Scaffold, Propagate, FindRepeats Steps to force:

[citation] MetAMOS Treangen, TJ ⇔ Koren, S, Sommer, DD, Liu, B, Astrovskaya, I, Ondov, B, Darling AE, Phillippy AM, Pop, M. MetAMOS: a modular and open source metagenomic assembly and analysis pipeline. Genome biology, 14(1), R2, 2013.

iMetAMOS Koren, S, Treangen, TJ, Hill, CM, Pop, M, Phillippy, AM. Automated ensemble assembly and validation of microbial genomes. bioRxiv preprint, 2014.

Step-specific configuration: [abundance] MetaPhyler /usr/local/metAMOS-1.5rc1/Utilities/cpp/Linux-x86_64 Liu B, Gibbons T, Ghodsi M, Treangen T, Pop M. Accurate and fast estimation of taxonomic profiles from metagenomic shotgun sequences. BMC Genomics. 2011;12 Suppl 2:S4. Epub 2011 Jul 27.

[multialign] M-GCAT /usr/local/metAMOS-1.5rc1/Utilities/cpp/Linux-x86_64 Treangen TJ, Messeguer X. M-GCAT: interactively and efficiently constructing large-scale multiple genome comparison frameworks in closely related species. BMC Bioinformatics, 2006.

[fannotate] BLAST /usr/local/metAMOS-1.5rc1/Utilities/cpp/Linux-x86_64 Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ. Basic local alignment search tool. J Mol Biol. 1990 Oct 5;215(3):403-10.

[scaffold] Bambus 2 /usr/local/metAMOS-1.5rc1/AMOS/Linux-x86_64/bin Koren S, Treangen TJ, Pop M. Bambus 2: scaffolding metagenomes. Bioinformatics 27(21): 2964-2971 2011.

[findorfs] Prokka /usr/local/metAMOS-1.5rc1/Utilities/cpp/Linux-x86_64/prokka/bin Prokka: Prokaryotic Genome Annotation System - http://vicbioinformatics.com/

[annotate] Kraken /usr/local/metAMOS-1.5rc1/Utilities/cpp/Linux-x86_64/kraken/bin Wood DE, Salzberg SL. Rapid phylogenetic sequence classification through repeated exact alignment. In preparation.

[assemble] ABySS /usr/local/metAMOS-1.5rc1/Utilities/cpp/Linux-x86_64/abyss/bin Simpson, JT, Wong, K, Jackman, SD, Schein, JE, Jones, SJ, Birol, Ä°. ABySS: a parallel assembler for short read sequence data. Genome research, 19(6), 1117-1123, 2009.

Velvet /usr/local/metAMOS-1.5rc1/Utilities/cpp/Linux-x86_64/velvet Zerbino DR, Birney E. Velvet: algorithms for de novo short read assembly using de Bruijn graphs. Genome Res. 2008 May;18(5):821-9.

SOAPdenovo2

Luo, R, Liu, B, Xie, Y, Li, Z, Huang, W, Yuan, J, He G, Chen Y, Pan Q, Liu Y, Tang J, Wu G, Zhang H, Shi Y, Liu Y, Yu C, Wang B, Lu Y, Han C, Cheung DW, Yiu S, Peng S, Xiaoqian Z, Liu G, Liao X, Li Y, Yang H, Wang J, Lam T, Wang J. SOAPdenovo2: an empirically improved memory-efficient short-read de novo assembler. GigaScience, 1(1), 18, 2012.

SGA /usr/local/metAMOS-1.5rc1/Utilities/cpp/Linux-x86_64/sga/bin Simpson, JT, Durbin, R. Efficient de novo assembly of large genomes using compressed data structures. Genome Research, 22(3), 549-556, 2012.

SPAdes /usr/local/metAMOS-1.5rc1/Utilities/cpp/Linux-x86_64/spades/bin Anton Bankevich, Sergey Nurk, Dmitry Antipov, Alexey A. Gurevich, Mikhail Dvorkin, Alexander S. Kulikov, Valery M. Lesin, Sergey I. Nikolenko, Son Pham, Andrey D. Prjibelski, Alexey V. Pyshkin, Alexander V. Sirotkin, Nikolay Vyahhi, Glenn Tesler, Max A. Alekseyev, and Pavel A. Pevzner. Journal of Computational Biology. May 2012, 19(5): 455-477. doi:10.1089/cmb.2012.0021.

MaSuRCA

Zimin, A, Marçais, G, Puiu, D, Roberts, M, Salzberg, SL, Yorke, JA. The MaSuRCA genome assembler. Bioinformatics, btt476, 2013.

[mapreads] Bowtie /usr/local/metAMOS-1.5rc1/Utilities/cpp/Linux-x86_64 Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efficient alignment of short DNA sequences to the human genome. Genome Biol. 2009;10(3):R25. Epub 2009 Mar 4.

[preprocess] metAMOS built-in filtering N/A

[validate] FRCbam

Vezzi, F, Narzisi, G, Mishra, B. Reevaluating assembly evaluations with feature response curves: GAGE and assemblathons. PloS ONE, 7(12), e52210, 2013.

CGAL /usr/local/metAMOS-1.5rc1/Utilities/cpp/Linux-x86_64/cgal Rahman, A, Pachter, L CGAL: computing genome assembly likelihoods. Genome biology, 14(1), R8, 2013.

ALE /usr/local/metAMOS-1.5rc1/Utilities/cpp/Linux-x86_64/ALE/src Clark, SC, Egan, R, Frazier, PI, Wang, Z. ALE: a generic assembly likelihood evaluation framework for assessing the accuracy of genome and metagenome assemblies. Bioinformatics, 29(4) 435-443, 2013.

QUAST /usr/local/metAMOS-1.5rc1/quast Gurevich, A, Saveliev, V, Vyahhi, N, Tesler, G. QUAST: quality assessment tool for genome assemblies. Bioinformatics, 29(8), 1072-1075, 2013.

Prokka /usr/local/metAMOS-1.5rc1/Utilities/cpp/Linux-x86_64/prokka/bin Prokka: Prokaryotic Genome Annotation System - http://vicbioinformatics.com/

FreeBayes /usr/local/metAMOS-1.5rc1/Utilities/cpp/Linux-x86_64/freebayes/bin Garrison, E, Marth, G. Haplotype-based variant detection from short-read sequencing. arXiv preprint arXiv:1207.3907, 2012.

LAP /usr/local/metAMOS-1.5rc1/LAP Ghodsi M, Hill CM, Astrovskaya I, Lin H, Sommer DD, Koren S, Pop M. De novo likelihood-based measures for comparing genome assemblies. BMC research notes 6:334, 2013.

REAPR

Hunt, M, Kikuchi, T, Sanders, M, Newbold, C, Berriman, M, & Otto, TD. REAPR: a universal tool for genome assembly evaluation. Genome biology, 14(5), R47, 2013.

[other] Krona /usr/local/metAMOS-1.5rc1/KronaTools/bin Ondov BD, Bergman NH, Phillippy AM. Interactive metagenomic visualization in a Web browser. BMC Bioinformatics. 2011 Sep 30;12:385.

KmerGenie /usr/local/metAMOS-1.5rc1/Utilities/cpp/Linux-x86_64/kmergenie Chikhi, R, Medvedev, P. Informed and Automated k-Mer Size Selection for Genome Assembly. Bioinformatics btt310, 2013.

Starting Task = preprocess.PREPROCESS Warning: could not auto-pick a kmer, KmerGenie not found. Defaulting to 31 Starting Task = assemble.ASSEMBLE Error: requested to run MaSuRCA (runSRCA.pl) but not available in specified location . Please check your specification and try again Error: Soapdenovo2 not found in . Please check your path and try again.

* MetAMOS Warning: soapdenovo2 assembler did not run successfully! * MetAMOS Warning: masurca assembler did not run successfully! Starting Task = mapreads.MAPREADS Starting Task = findorfs.FINDORFS

Starting Task = validate.VALIDATE * metAMOS: Error, type FRCBAM is not available, skipping * metAMOS: Error, type REAPR is not available, skipping Warning: selected score FRCBAM was not available, skipping it Warning: selected score REAPR was not available, skipping it recruiting genomes.. done! recruited 4134 genomes!

Starting Task = findrepeats.FINDREPEATS Starting Task = annotate.ANNOTATE Starting Task = functionalannotation.FUNCTIONALANNOTATION Starting Task = scaffold.SCAFFOLD

Starting Task = findscaffoldorfs.FINDSCAFFOLDORFS Starting Task = abundance.ABUNDANCE Starting Task = propagate.PROPAGATE Starting Task = classify.CLASSIFY Starting Task = postprocess.POSTPROCESS Oops, MetAMOS finished with errors! see text in red above for details.

skoren commented 10 years ago

Hi,

First the installation of SOAP2, MaSuRCA, and KmerGenie. MaSuRCA requires GCC 4.4 or newer on your system so if it was not available, it will not be installed. KmerGenie requires a working installation of R so if that is not available, it will not be installed. You can use metAMOS without having those tools installed. For SOAP2, if you can provide the output from your run of INSTALL.py, it should document why it was not installed.

Your first two issues are due to skipping the Assemble and MapReads steps. Those steps are currently required and cannot be skipped. The Assemble step can be implicitly skipped by specifying input contigs to init pipeline.I've updated the code to disable skipping these steps.

For the last two errors (GAGE-B and iMetAMOS), I would recommend upgrading to metAMOS v1.5rc2. It has several bug fixes that could address your issues and will provide more detailed logging of the error in verbose mode to allow us to identify the error.

kittim commented 10 years ago

Thank you Dr. Koren, I will not skip Assembly and MapReads.

Since it may be a while before I can have my administrator update to metAMOS v1.5rc2, may I address one more issue that seems rather consistent to see if it falls into cases you've encountered before? When I have metAMOS go as further down the pipeline as possible, I see it pausing somewhere in Postprocess, right after statistics.pl. It is not "finishing with error"; the program looks stalled at that point. (process still running)

I put below my commands, end of output, Logs/COMMANDS.log and Logs/POSTPROCESS.log, as well as the contents of Postprocess/out and its html subdirectory. It shows asmstats.out in Postprocess/out/html. covstats.out and stats.out are not there. I also see warning about 2D hist being seemly transposed.

Thank you again.

/// Commands /// DIR/initPipeline -1 tmp1.fastq -2 tmp2.fastq -d outDIR -i 300:500 DIR/runPipeline -a velvet -z Family -d outDIR -p 8 -v

/// tail output /// *\ metAMOS running command: ln /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/abundance.krona.html /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html/Abundance.html

*\ metAMOS running command: touch /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/ref.name

** metAMOS running command: mv /home/skim/metAMOS/READ_00/outDIR/Preprocess/out/.fastqc /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html

*\ metAMOS running command: unlink /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html/propagate.in.clusters

**/usr/lib64/python2.6/site-packages/matplotlib/axes.py:7066: UserWarning: 2D hist should be nsamples x nvariables; this looks transposed warnings.warn('2D hist should be nsamples x nvariables; '

/// tail Logs/COMMANDS.log /// |2014-04-23 13:02:55| ln /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/abundance.krona.html /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html/Abundance.html |2014-04-23 13:02:55| touch /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/ref.name |2014-04-23 13:02:55| mv /home/skim/metAMOS/READ_00/outDIR/Preprocess/out/*.fastqc /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html |2014-04-23 13:02:55| unlink /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html/propagate.in.clusters |2014-04-23 13:02:55| ln /home/skim/metAMOS/READ_00/outDIR/Propagate/in/proba.clusters /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html/propagate.in.clusters |2014-04-23 13:02:55| unlink /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html/propagate.out.clusters |2014-04-23 13:02:55| ln /home/skim/metAMOS/READ_00/outDIR/Propagate/out/proba.clusters /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html/propagate.out.clusters |2014-04-23 13:02:55| unlink /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html/FunctionalAnnotation.html |2014-04-23 13:02:55| ln /home/skim/metAMOS/READ_00/outDIR/FunctionalAnnotation/out/ec.krona.html /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html/FunctionalAnnotation.html |2014-04-23 13:02:56| perl -I /usr/local/metAMOS-1.5rc1/AMOS/Linux-x86_64/lib /usr/local/metAMOS-1.5rc1/Utilities/perl/statistics.pl /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/proba.scf.fa > /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html/asmstats.out

/// tail Logs/POSTPROCESS.log /// unlink: cannot unlink /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/proba.ctg.cvg': No such file or directory unlink: cannot unlink/home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html': No such file or directory unlink: cannot unlink /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html/family.classified': No such file or directory unlink: cannot unlink/home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html/Annotate.html': No such file or directory unlink: cannot unlink /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html/Abundance.html': No such file or directory mv: cannot stat/home/skim/metAMOS/READ_00/outDIR/Preprocess/out/*.fastqc': No such file or directory unlink: cannot unlink /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html/propagate.in.clusters': No such file or directory unlink: cannot unlink/home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html/propagate.out.clusters': No such file or directory unlink: cannot unlink /home/skim/metAMOS/READ_00/outDIR/Postprocess/out/html/FunctionalAnnotation.html': No such file or directory ln: accessing/home/skim/metAMOS/READ_00/outDIR/FunctionalAnnotation/out/ec.krona.html': No such file or directory

/// ls -l Postprocess/out /// total 84412 -rw-rw----. 3 skim skim 2451 Apr 23 12:58 abundance.krona.html -rw-rw----. 3 skim skim 149667 Apr 23 12:53 annotate.krona.html -rw-rw----. 2 skim skim 0 Apr 23 12:51 asm.scores -rw-rw----. 2 skim skim 9 Apr 23 12:51 best.asm lrwxrwxrwx. 2 skim skim 46 Apr 23 13:02 family.classified -> /home/skim/metAMOS/READ_00/outDIR/Classify/out -rw-rw----. 3 skim skim 411077 Apr 23 12:53 family.original.annots -rw-rw----. 2 skim skim 4946064 Apr 23 12:53 family.original.reads.annots -rw-rw----. 1 skim skim 677595 Apr 23 13:02 family.propagated.annots -rw-rw----. 1 skim skim 6645443 Apr 23 13:02 family.propagated.reads.annots drwxrwx---. 2 skim skim 4096 Apr 24 06:56 html lrwxrwxrwx. 1 skim skim 55 Apr 23 13:02 proba.bnk -> /home/skim/metAMOS/READ_00/outDIR/Scaffold/in/proba.bnk -rw-rw----. 1 skim skim 3643 Apr 23 13:02 proba.classify.txt -rw-rw----. 3 skim skim 568635 Apr 23 12:49 proba.ctg.cnt -rw-rw----. 3 skim skim 608900 Apr 23 12:49 proba.ctg.cvg -rw-rw----. 1 skim skim 6756340 Apr 23 13:02 proba.ctg.fa -rw-rw----. 3 skim skim 2307107 Apr 23 12:53 proba.hits -rw-rw----. 3 skim skim 18803263 Apr 23 12:49 proba.lib1.contig.reads -rw-rw----. 3 skim skim 26972102 Apr 23 12:49 proba.lib1.unaligned.fasta -rw-rw----. 1 skim skim 0 Apr 23 13:02 proba.motifs.fa -rw-rw----. 5 skim skim 3246256 Apr 23 12:53 proba.orf.faa -rw-rw----. 5 skim skim 7387926 Apr 23 12:52 proba.orf.fna -rw-rw----. 1 skim skim 6352665 Apr 23 13:02 proba.scf.fa -rw-rw----. 1 skim skim 0 Apr 23 13:02 ref.name

/// ls -l Postprocess/out/html /// total 27468 -rw-rw----. 3 skim skim 2451 Apr 23 12:58 Abundance.html -rw-rw----. 3 skim skim 149667 Apr 23 12:53 Annotate.html -rw-rw----. 1 skim skim 306 Apr 23 13:02 asmstats.out lrwxrwxrwx. 2 skim skim 46 Apr 23 13:02 family.classified -> /home/skim/metAMOS/READ_00/outDIR/Classify/out -rw-rw----. 1 skim skim 26415233 Apr 23 17:18 hist_contigs.pdf -rw-rw----. 1 skim skim 34125 Apr 23 16:55 hist_contigs.png -rw-rw----. 1 skim skim 65 Apr 23 13:02 plot.tab -rw-rw----. 2 skim skim 411077 Apr 23 12:58 propagate.in.clusters -rw-rw----. 2 skim skim 677595 Apr 23 12:58 propagate.out.clusters

treangen commented 10 years ago

hi kittim,

I just wanted to quickly reply to your comment:

Since it may be a while before I can have my administrator update to metAMOS v1.5rc2,

We also provide a single file binary installation that would allow you to download and use v1.5rc2 while waiting for your update. Here are more details:

http://metamos.readthedocs.org/en/latest/content/frozenbinary.html#metamos-pyinstaller-single-file-binary

Please note that the frozen binary has reduced functionality/features compared to installing from source, but would allow you to use the latest release (v1.5rc2).

-Todd

skoren commented 10 years ago

The most likely causes of your error in Postprocess is either a missing Statistics library for perl. Normally this is installed by INSTALL.py but if something went wrong when your sysadmin installed it or if it was installed on a different system than where you are running it, it could be missing. You can manually install Statistics::Descriptive for perl or check if it is available using: perl -MStatistics::Descriptive -e 0 && echo $? If it outputs 0 then the package is installed and available.

The other possibility is a bug in RC1 fixed in RC2, issue #118 and there is a workaround documented in that issue that may help fix your error. Generally though I would recommend using rc2 or the frozen binary as those will also provide us more information to diagnose errors.

kittim commented 10 years ago

Thank you all; I do see Statistics/Descriptive file, though. If I try perl -MStatistics::Descriptive -e 0 && echo $, I get $.

skoren commented 10 years ago

It should be perl -MStatistics::Descriptive -e 0 && echo $? with a ? on the end. Given that you have the file this is likely not the issue and #118 fixed in RC2 is the more likely cause. Can you try the RC2 frozen binary and see if your iMetAMOS test runs there?

treangen commented 9 years ago

Closing due to lack of activity, issue likely cause by a lack of Statistics perl library.