search

marbl / metAMOS

A metagenomic and isolate assembly and analysis pipeline built with AMOS
http://marbl.github.io/metAMOS
Other
93 stars 45 forks source link

run_pipeline_test.sh Fails #181

Open jonathanjacobs opened 9 years ago

jonathanjacobs commented 9 years ago

Just installed metamos on a new system - and the test suite is failing with an error in matplotlib .

note: I recently removed the latest version of matplotlib and downgraded to v1.3.1, as recommend by the metamos install, 

[jjacobs metAMOS-1.5rc3]$ python INSTALL.py core
<<Welcome to metAMOS install>>
Selected to install workflowName: CORE.
Selected to install workflowName: IMETAMOS.
Will automatically install:
    Pysam
    Fastqc
    Ca
    Ale
    Sga
    Velvet-Sc
    Ray
    Cython
    Metavelvet
    Velvet
    Soap2
    Frcbam
    Quast
    Eautils
    Freebayes
    Numpy
    Edena
    Uniprot
    Abyss
    Cgal
    Mira
    Matplotlib
    Prokka
    Idba
    Cmake
    Lap
    Masurca
    Kronatools
    Sra
    Spades
    Kraken
    Psutil
    Setuptools
    Amos
    Kmergenie
    Reapr
Run setup.py..
running install_scripts
running egg_info
writing metAMOS.egg-info/PKG-INFO
writing top-level names to metAMOS.egg-info/top_level.txt
writing dependency_links to metAMOS.egg-info/dependency_links.txt
reading manifest file 'metAMOS.egg-info/SOURCES.txt'
writing manifest file 'metAMOS.egg-info/SOURCES.txt'
running build_scripts
copying build/scripts-2.6/initPipeline.py -> /home/jjacobs/src/metAMOS-1.5rc3
copying build/scripts-2.6/runPipeline.py -> /home/jjacobs/src/metAMOS-1.5rc3
changing mode of /home/jjacobs/src/metAMOS-1.5rc3/initPipeline.py to 775
changing mode of /home/jjacobs/src/metAMOS-1.5rc3/runPipeline.py to 775
running build_ext
skipping './src/preprocess.c' Cython extension (up-to-date)
skipping './src/annotate.c' Cython extension (up-to-date)
skipping './src/fannotate.c' Cython extension (up-to-date)
skipping './src/mapreads.c' Cython extension (up-to-date)
skipping './src/findorfs.c' Cython extension (up-to-date)
skipping './src/findscforfs.c' Cython extension (up-to-date)
skipping './src/scaffold.c' Cython extension (up-to-date)
[jjacobs metAMOS-1.5rc3]$ cd Test/
[jjacobs Test]$ ./run_pipeline_test.sh 
Error: cannot find BLAST DB directory, expected it in /home/jjacobs/src/metAMOS-1.5rc3/Utilities/DB/. Disabling blastdb dependent programs
Project directory already exists, please specify another
Alternatively, use runPipeline to run an existing project
Error: cannot find BLAST DB directory, expected it in /home/jjacobs/src/metAMOS-1.5rc3/Utilities/DB/. Disabling blastdb dependent programs
Error: cannot find BLAST DB directory, expected it in /home/jjacobs/src/metAMOS-1.5rc3/Utilities/DB/. Disabling blastdb dependent programs
**no blast DB directory available, disabling steps requiring BLAST DB
[Steps to be skipped]:  set(['FunctionalAnnotation', 'FindRepeats'])
Starting Task = runpipeline.RUNPIPELINE
Starting metAMOS pipeline
Error: cannot find BLAST DB directory, expected it in /home/jjacobs/src/metAMOS-1.5rc3/Utilities/DB/. Disabling blastdb dependent programs
Warning: BLASR is not found, some functionality will not be available
Warning: Newbler is not found, some functionality will not be available
Warning: MetaGeneMark is not found, some functionality will not be available
Warning: SignalP+ is not found, some functionality will not be available
Warning: metaphylerClassify is not found, some functionality will not be available
Warning: PhyloSift was not found, will not be available

/home/jjacobs/src/metAMOS-1.5rc3/src/utils.py:1276: DeprecationWarning: psutil.cached_phymem is deprecated; use psutil.virtual_memory().cached instead
  cacheusage = psutil.cached_phymem()
/home/jjacobs/src/metAMOS-1.5rc3/src/utils.py:1277: DeprecationWarning: psutil.phymem_usage is deprecated
  memusage =  `psutil.phymem_usage()`.split(",")
[Available RAM: 60 GB]
    *ok

________________________________________
Tasks which will be run:

Task = assemble.SplitAssemblers
Task = assemble.Assemble
Task = assemble.CheckAsmResults
Task = assemble.SplitMappers
Task = mapreads.MapReads
Task = mapreads.CheckMapResults
Task = mapreads.SplitForORFs
Task = findorfs.FindORFS
Task = validate.Validate
Task = findreps.FindRepeats
Task = annotate.Annotate
Task = fannotate.FunctionalAnnotation
Task = scaffold.Scaffold
Task = findscforfs.FindScaffoldORFS
Task = abundance.Abundance
Task = propagate.Propagate
Task = classify.Classify
Task = postprocess.Postprocess
________________________________________
metAMOS configuration summary:
metAMOS Version:    v1.5rc2 "Praline Brownie"  workflows: core,imetamos
Time and Date:      2015-02-20
Working directory:  /home/jjacobs/src/metAMOS-1.5rc3/Test/test1
Prefix:         proba
K-Mer:          55
Threads:        15
Taxonomic level:    class
Verbose:        False
Steps to skip:      MultiAlign, FunctionalAnnotation, FindRepeats
Steps to force:     Abundance, FindORFS, Annotate, Propagate, MapReads, Assemble, FindScaffoldORFS, Classify

[citation]
MetAMOS
    Treangen, TJ ⇔  Koren, S, Sommer, DD, Liu, B, Astrovskaya, I, Ondov, B, Darling AE, Phillippy AM, Pop, M. MetAMOS: a modular and open source metagenomic assembly and analysis pipeline. Genome biology, 14(1), R2, 2013.

Step-specific configuration:
[abundance]
MetaPhyler

    Liu B, Gibbons T, Ghodsi M, Treangen T, Pop M. Accurate and fast estimation of taxonomic profiles from metagenomic shotgun sequences. BMC Genomics. 2011;12 Suppl 2:S4. Epub 2011 Jul 27.

[multialign]
M-GCAT
    /home/jjacobs/src/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64
    Treangen TJ, Messeguer X. M-GCAT: interactively and efficiently constructing large-scale multiple genome comparison frameworks in closely related species. BMC Bioinformatics, 2006.

[fannotate]
BLAST
    /home/jjacobs/src/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64
    Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ. Basic local alignment search tool. J Mol Biol. 1990 Oct 5;215(3):403-10.

[scaffold]
Bambus 2
    /home/jjacobs/src/metAMOS-1.5rc3/AMOS/Linux-x86_64/bin
    Koren S, Treangen TJ, Pop M. Bambus 2: scaffolding metagenomes. Bioinformatics 27(21): 2964-2971 2011.

[findorfs]
FragGeneScan
    /home/jjacobs/src/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64
    Rho M, Tang H, Ye Y: FragGeneScan: predicting genes in short and error-prone reads. Nucleic Acids Research 2010, 38:e191-e191.

[annotate]
Kraken
    /home/jjacobs/src/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64/kraken/bin
    Wood DE, Salzberg SL: Kraken: ultrafast metagenomic sequence classification using exact alignments. Genome Biology 2014, 15:R46.

[assemble]
SOAPdenovo
    /home/jjacobs/src/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64
    Li Y, Hu Y, Bolund L, Wang J: State of the art de novo assembly of human genomes from massively parallel sequencing data.Human genomics 2010, 4:271-277.

[mapreads]
Bowtie
    /home/jjacobs/src/metAMOS-1.5rc3/Utilities/cpp/Linux-x86_64
    Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efficient alignment of short DNA sequences to the human genome. Genome Biol. 2009;10(3):R25. Epub 2009 Mar 4.

[preprocess]
metAMOS built-in filtering
    N/A

[validate]
LAP
    /home/jjacobs/src/metAMOS-1.5rc3/LAP
    Ghodsi M, Hill CM, Astrovskaya I, Lin H, Sommer DD, Koren S, Pop M. De novo likelihood-based measures for comparing genome assemblies. BMC research notes 6:334, 2013.

[other]
Krona
    /home/jjacobs/src/metAMOS-1.5rc3/KronaTools/bin
    Ondov BD, Bergman NH, Phillippy AM. Interactive metagenomic visualization in a Web browser. BMC Bioinformatics. 2011 Sep 30;12:385.

Starting Task = assemble.ASSEMBLE
    Job  = [preprocess.success -> *.run] completed
Completed Task = assemble.SplitAssemblers
    Job  = [soapdenovo.55.run -> soapdenovo.55.asm.contig] completed
Completed Task = assemble.Assemble
    Job  = [[soapdenovo.55.asm.contig] -> [assemble.ok]] completed
Completed Task = assemble.CheckAsmResults
Uptodate Task = assemble.SplitMappers
Starting Task = mapreads.MAPREADS
    Job  = [soapdenovo.55.asm.contig -> soapdenovo.55.contig.cvg] completed
Completed Task = mapreads.MapReads
    Job  = [[soapdenovo.55.contig.cvg] -> [mapreads.ok]] completed
Completed Task = mapreads.CheckMapResults
Uptodate Task = mapreads.SplitForORFs
Starting Task = findorfs.FINDORFS
    Job  = [soapdenovo.55.contig.cvg -> soapdenovo.55.faa] completed
Completed Task = findorfs.FindORFS
Starting Task = validate.VALIDATE
    Job  = [[soapdenovo.55.faa] -> [validate.ok]] completed
Completed Task = validate.Validate
Starting Task = findrepeats.FINDREPEATS
    Job  = [proba.fna -> proba.repeats] completed
Completed Task = findreps.FindRepeats
Starting Task = annotate.ANNOTATE
    Job  = [proba.faa -> proba.hits] completed
Completed Task = annotate.Annotate
Starting Task = functionalannotation.FUNCTIONALANNOTATION
    Job  = [proba.faa -> [blast.out, krona.ec.input]] completed
Completed Task = fannotate.FunctionalAnnotation
Starting Task = scaffold.SCAFFOLD
    Job  = [[proba.asm.contig] -> scaffold.ok] completed
Completed Task = scaffold.Scaffold
Starting Task = findscaffoldorfs.FINDSCAFFOLDORFS
    Job  = [proba.linearize.scaffolds.final -> proba.fna] completed
Completed Task = findscforfs.FindScaffoldORFS
Starting Task = abundance.ABUNDANCE
    Job  = [proba.asm.contig -> proba.taxprof.pct.txt] completed
Completed Task = abundance.Abundance
Starting Task = propagate.PROPAGATE
    Job  = [proba.annots -> propagate.ok] completed
Completed Task = propagate.Propagate
Starting Task = classify.CLASSIFY
    Job  = [proba.clusters -> classify.ok] completed
Completed Task = classify.Classify
Starting Task = postprocess.POSTPROCESS
/usr/lib64/python2.6/site-packages/matplotlib/axes.py:7066: UserWarning: 2D hist should be nsamples x nvariables; this looks transposed
  warnings.warn('2D hist should be nsamples x nvariables; '
Oops, MetAMOS finished with errors! see text in red above for details.
[jjacobs Test]$ python -V
Python 2.7.8

any ideas? hopefully a quick fix I'm simply overlooking.

treangen commented 9 years ago

hi Jonathan,

Looks like the warning/error is due to matplotlib version incompatibility. Are you still able to view/open the final HTML summary plot? It might be available, but if not please let me know. Either way, all generated output will still be available in the ./Postprocess/out directory.

Regarding the warning/error, my best guess is that you installed/downgraded matplotlib 1.3.1 for python 2.7, yet a newer matplotlib version installed for python 2.6 is getting used, triggering the known issue:

/usr/lib64/python2.6/site-packages/matplotlib/axes.py

and

Python 2.7.8

I will work on committing a fix for v1.5rc3 to address the matplotlib version issue when creating the HTML plot assembly histograms. Once available, a simple git pull should fix it, assuming you cloned from the v1.5rc3 release.

In the meantime, a potential quick fix would be to edit the first lines of initPipeline & runPipeline to point to your python 2.7.8, instead of 2.6, assuming 2.6 is indeed getting used.

-Todd