search

marbl / metAMOS

A metagenomic and isolate assembly and analysis pipeline built with AMOS
http://marbl.github.io/metAMOS
Other
93 stars 45 forks source link

Could not import psutil, disabling - runPipeline #90

Closed metamoszhang closed 10 years ago

metamoszhang commented 10 years ago

Hi,

I am using a frozen binary MetAMOS with all BLAST DBs. But when I test runPipeline command, an error "Could not import psutil, disabling." was reported. Now, I used Python 2.7.3 and tried to install psutil-0.5.1, psutil 0.6.1, and psutil-1.1.3. But the error was always reported. Please help me resolve it.

My python: [zzg@fat-node-1 ~]$ python Python 2.7.3 (default, Jun 6 2013, 17:48:01) [GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

psutil install:

[zzg@fat-node-1 psutil-0.5.1]$ python setup.py install running install running bdist_egg running egg_info writing psutil.egg-info/PKG-INFO writing top-level names to psutil.egg-info/top_level.txt writing dependency_links to psutil.egg-info/dependencylinks.txt reading manifest file 'psutil.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '.html' under directory 'docs' warning: no files found matching '_.dia' under directory 'docs' warning: no files found matching '*.png' under directory 'docs' writing manifest file 'psutil.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/psutil copying psutil/_psposix.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/_common.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/init.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/_psmswindows.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/_pslinux.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/error.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/_psosx.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/_psbsd.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/_compat.py -> build/lib.linux-x86_64-2.7/psutil running build_ext building '_psutil_linux' extension creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/psutil gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o -o build/lib.linux-x86_64-2.7/_psutil_linux.so building '_psutil_posix' extension gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/_psutil_posix.so creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/_psutil_posix.so -> build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/_psutil_linux.so -> build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/_psposix.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/_common.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/init.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/_psmswindows.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/_pslinux.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/error.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/_psosx.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/_psbsd.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/_compat.py -> build/bdist.linux-x86_64/egg/psutil byte-compiling build/bdist.linux-x86_64/egg/psutil/_psposix.py to _psposix.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/_common.py to _common.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/_psmswindows.py to _psmswindows.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/_pslinux.py to _pslinux.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/error.py to error.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/_psosx.py to _psosx.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/_psbsd.py to _psbsd.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/_compat.py to _compat.pyc creating stub loader for _psutil_linux.so creating stub loader for _psutil_posix.so byte-compiling build/bdist.linux-x86_64/egg/_psutil_linux.py to _psutil_linux.pyc byte-compiling build/bdist.linux-x86_64/egg/_psutil_posix.py to _psutil_posix.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt zip_safe flag not set; analyzing archive contents... creating dist creating 'dist/psutil-0.5.1-py2.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing psutil-0.5.1-py2.7-linux-x86_64.egg Copying psutil-0.5.1-py2.7-linux-x86_64.egg to /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages Removing psutil 0.6.1 from easy-install.pth file Adding psutil 0.5.1 to easy-install.pth file

Installed /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages/psutil-0.5.1-py2.7-linux-x86_64.egg Processing dependencies for psutil==0.5.1 Finished processing dependencies for psutil==0.5.1

runPipeline test:

[zzg@fat-node-1 ~]$ runPipeline -h Could not import psutil, disabling. Could not import psutil, disabling. usage: runPipeline [options] -d projectdir -h = : print help [this message] -j = : just output all of the programs and citations then exit (default = NO) -v = : verbose output? (default = NO) -d = : directory created by initPipeline (REQUIRED)

[options]: [pipeline_opts] [misc_opts]

[pipeline_opts]: options that affect the pipeline execution Pipeline consists of the following steps: Preprocess, Assemble, FindORFS, MapReads, Abundance, Annotate, FunctionalAnnotation, Scaffold, Propagate, Classify, Postprocess Each of these steps can be referred to by the following options: -f = : force this step to be run (default = NONE) -s = : start at this step in the pipeline (default = Preprocess) -e = : end at this step in the pipeline (default = Postprocess) -n = : step to skip in pipeline (default=NONE)

For each step you can fine-tune the execution as follows [Preprocess] -t = : filter input reads? (default = NO) -q = : produce FastQC quality report for reads with quality information (fastq or sff)? (default = NO) [Assemble] -a = : genome assembler to use (default = SOAPdenovo) -k = : k-mer size to be used for assembly (default = 31) -o = >: min overlap length [MapReads] -m = : read mapper to use? (default = bowtie) -i = : save bowtie (i)ndex? (default = NO) -b = : create library specific per bp coverage of assembled contigs (default = NO) [FindORFS] -g = : gene caller to use (default=FragGeneScan) -l = : min contig length to use for ORF call (default = 300) -x = >: min contig coverage to use for ORF call (default = 3X) [Annotate] -c = : classifier to use for annotation (default = FCP) -u = : annotate unassembled reads? (default = NO) [Classify] -z = : taxonomic level to categorize at (default = class)

[misc_opts]: Miscellaneous options -B = : blast DBs not available (default = NO) -r = : retain the AMOS bank? (default = NO) -p = : number of threads to use (be greedy!) (default=1) -4 = : 454 data? (default = NO) -L = : generate local Krona plots. Local Krona plots can only be viewed on the machine they are generated on but will work on a system with no internet connection (default = NO) [zzg@fat-node-1 ~]$

skoren commented 10 years ago

Hi,

Psutil should be included in the binary release but this could be a system-specific issue causing the warning. Installing psutils on your system python will not change the version of python/packages being used by the binary release since it is fully self-contained. However, psutil is only used to estimate available CPU/memory so as long as you specify the number of cpus to use for the pipeline and have at least 32GB of ram available, you should be able to use metAMOS without issues.

Sergey

On Nov 12, 2013, at 10:24 PM, metamoszhang notifications@github.com wrote:

Hi,

I am using a frozen binary MetAMOS with all BLAST DBs. But when I test runPipeline command, an error "Could not import psutil, disabling." was reported. Now, I used Python 2.7.3 and tried to install psutil-0.5.1, psutil 0.6.1, and psutil-1.1.3. But the error was always reported. Please help me resolve it.

My python: [zzg@fat-node-1 ~]$ python Python 2.7.3 (default, Jun 6 2013, 17:48:01) [GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

psutil install:

[zzg@fat-node-1 psutil-0.5.1]$ python setup.py install running install running bdist_egg running egg_info writing psutil.egg-info/PKG-INFO writing top-level names to psutil.egg-info/top_level.txt writing dependency_links to psutil.egg-info/dependency_links.txt reading manifest file 'psutil.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '.html' under directory 'docs' warning: no files found matching '.dia' under directory 'docs' warning: no files found matching '*.png' under directory 'docs' writing manifest file 'psutil.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/psutil copying psutil/psposix.py -> build/lib.linux-x8664-2.7/psutil copying psutil/common.py -> build/lib.linux-x8664-2.7/psutil copying psutil/init.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psmswindows.py -> build/lib.linux-x8664-2.7/psutil copying psutil/pslinux.py -> build/lib.linux-x8664-2.7/psutil copying psutil/error.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psosx.py -> build/lib.linux-x8664-2.7/psutil copying psutil/psbsd.py -> build/lib.linux-x8664-2.7/psutil copying psutil/compat.py -> build/lib.linux-x8664-2.7/psutil running build_ext building 'psutillinux' extension creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/psutil gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutillinux.c -o build/temp.linux-x86_64-2.7/psutil/psutillinux.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutillinux.o -o build/lib.linux-x86_64-2.7/psutillinux.so building 'psutilposix' extension gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutilposix.c -o build/temp.linux-x86_64-2.7/psutil/psutilposix.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutilposix.o -o build/lib.linux-x86_64-2.7/psutilposix.so creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutilposix.so -> build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutillinux.so -> build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psposix.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/common.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/init.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psmswindows.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/pslinux.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/error.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psosx.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psbsd.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/compat.py -> build/bdist.linux-x8664/egg/psutil byte-compiling build/bdist.linux-x86_64/egg/psutil/psposix.py to _psposix.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/common.py to _common.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psmswindows.py to _psmswindows.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/pslinux.py to _pslinux.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/error.py to error.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psosx.py to _psosx.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/psbsd.py to _psbsd.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/compat.py to _compat.pyc creating stub loader for _psutillinux.so creating stub loader for psutilposix.so byte-compiling build/bdist.linux-x86_64/egg/psutillinux.py to psutillinux.pyc byte-compiling build/bdist.linux-x86_64/egg/psutilposix.py to psutilposix.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt zip_safe flag not set; analyzing archive contents... creating dist creating 'dist/psutil-0.5.1-py2.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing psutil-0.5.1-py2.7-linux-x86_64.egg Copying psutil-0.5.1-py2.7-linux-x86_64.egg to /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages Removing psutil 0.6.1 from easy-install.pth file Adding psutil 0.5.1 to easy-install.pth file

Installed /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages/psutil-0.5.1-py2.7-linux-x86_64.egg Processing dependencies for psutil==0.5.1 Finished processing dependencies for psutil==0.5.1

runPipeline test:

[zzg@fat-node-1 ~]$ runPipeline -h Could not import psutil, disabling. Could not import psutil, disabling. usage: runPipeline [options] -d projectdir -h = : print help [this message] -j = : just output all of the programs and citations then exit (default = NO) -v = : verbose output? (default = NO) -d = : directory created by initPipeline (REQUIRED)

[options]: [pipeline_opts] [misc_opts]

[pipeline_opts]: options that affect the pipeline execution Pipeline consists of the following steps: Preprocess, Assemble, FindORFS, MapReads, Abundance, Annotate, FunctionalAnnotation, Scaffold, Propagate, Classify, Postprocess Each of these steps can be referred to by the following options: -f = : force this step to be run (default = NONE) -s = : start at this step in the pipeline (default = Preprocess) -e = : end at this step in the pipeline (default = Postprocess) -n = : step to skip in pipeline (default=NONE)

For each step you can fine-tune the execution as follows [Preprocess] -t = : filter input reads? (default = NO) -q = : produce FastQC quality report for reads with quality information (fastq or sff)? (default = NO) [Assemble] -a = : genome assembler to use (default = SOAPdenovo) -k = : k-mer size to be used for assembly (default = 31) -o = >: min overlap length [MapReads] -m = : read mapper to use? (default = bowtie) -i = : save bowtie (i)ndex? (default = NO) -b = : create library specific per bp coverage of assembled contigs (default = NO) [FindORFS] -g = : gene caller to use (default=FragGeneScan) -l = : min contig length to use for ORF call (default = 300) -x = >: min contig coverage to use for ORF call (default = 3X) [Annotate] -c = : classifier to use for annotation (default = FCP) -u = : annotate unassembled reads? (default = NO) [Classify] -z = : taxonomic level to categorize at (default = class)

[misc_opts]: Miscellaneous options -B = : blast DBs not available (default = NO) -r = : retain the AMOS bank? (default = NO) -p = : number of threads to use (be greedy!) (default=1) -4 = : 454 data? (default = NO) -L = : generate local Krona plots. Local Krona plots can only be viewed on the machine they are generated on but will work on a system with no internet connection (default = NO) [zzg@fat-node-1 ~]$

— Reply to this email directly or view it on GitHub.

metamoszhang commented 10 years ago

Hi Sergey ,

Many thanks for your response. I got it. Now, I want to know which python version has been included binary release. My python is 2.7.3. I think, my python should no problem for running metAMOS.

Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-65199268 Fax: 86-871-65199318

From: Sergey Koren Date: 2013-11-14 09:31 To: treangen/metAMOS CC: metamoszhang Subject: Re: [metAMOS] Could not import psutil, disabling - runPipeline (#90) Hi,

Psutil should be included in the binary release but this could be a system-specific issue causing the warning. Installing psutils on your system python will not change the version of python/packages being used by the binary release since it is fully self-contained. However, psutil is only used to estimate available CPU/memory so as long as you specify the number of cpus to use for the pipeline and have at least 32GB of ram available, you should be able to use metAMOS without issues.

Sergey

On Nov 12, 2013, at 10:24 PM, metamoszhang notifications@github.com wrote:

Hi,

I am using a frozen binary MetAMOS with all BLAST DBs. But when I test runPipeline command, an error "Could not import psutil, disabling." was reported. Now, I used Python 2.7.3 and tried to install psutil-0.5.1, psutil 0.6.1, and psutil-1.1.3. But the error was always reported. Please help me resolve it.

My python: [zzg@fat-node-1 ~]$ python Python 2.7.3 (default, Jun 6 2013, 17:48:01) [GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

psutil install:

[zzg@fat-node-1 psutil-0.5.1]$ python setup.py install running install running bdist_egg running egg_info writing psutil.egg-info/PKG-INFO writing top-level names to psutil.egg-info/top_level.txt writing dependency_links to psutil.egg-info/dependency_links.txt reading manifest file 'psutil.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '.html' under directory 'docs' warning: no files found matching '.dia' under directory 'docs' warning: no files found matching '*.png' under directory 'docs' writing manifest file 'psutil.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/psutil copying psutil/psposix.py -> build/lib.linux-x8664-2.7/psutil copying psutil/common.py -> build/lib.linux-x8664-2.7/psutil copying psutil/init.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psmswindows.py -> build/lib.linux-x8664-2.7/psutil copying psutil/pslinux.py -> build/lib.linux-x8664-2.7/psutil copying psutil/error.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psosx.py -> build/lib.linux-x8664-2.7/psutil copying psutil/psbsd.py -> build/lib.linux-x8664-2.7/psutil copying psutil/compat.py -> build/lib.linux-x8664-2.7/psutil running build_ext building 'psutillinux' extension creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/psutil gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutillinux.c -o build/temp.linux-x86_64-2.7/psutil/psutillinux.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutillinux.o -o build/lib.linux-x86_64-2.7/psutillinux.so building 'psutilposix' extension gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutilposix.c -o build/temp.linux-x86_64-2.7/psutil/psutilposix.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutilposix.o -o build/lib.linux-x86_64-2.7/psutilposix.so creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutilposix.so -> build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutillinux.so -> build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psposix.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/common.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/init.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psmswindows.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/pslinux.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/error.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psosx.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psbsd.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/compat.py -> build/bdist.linux-x8664/egg/psutil byte-compiling build/bdist.linux-x86_64/egg/psutil/psposix.py to _psposix.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/common.py to _common.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psmswindows.py to _psmswindows.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/pslinux.py to _pslinux.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/error.py to error.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psosx.py to _psosx.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/psbsd.py to _psbsd.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/compat.py to _compat.pyc creating stub loader for _psutillinux.so creating stub loader for psutilposix.so byte-compiling build/bdist.linux-x86_64/egg/psutillinux.py to psutillinux.pyc byte-compiling build/bdist.linux-x86_64/egg/psutilposix.py to psutilposix.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt zip_safe flag not set; analyzing archive contents... creating dist creating 'dist/psutil-0.5.1-py2.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing psutil-0.5.1-py2.7-linux-x86_64.egg Copying psutil-0.5.1-py2.7-linux-x86_64.egg to /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages Removing psutil 0.6.1 from easy-install.pth file Adding psutil 0.5.1 to easy-install.pth file

Installed /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages/psutil-0.5.1-py2.7-linux-x86_64.egg Processing dependencies for psutil==0.5.1 Finished processing dependencies for psutil==0.5.1

runPipeline test:

[zzg@fat-node-1 ~]$ runPipeline -h Could not import psutil, disabling. Could not import psutil, disabling. usage: runPipeline [options] -d projectdir -h = : print help [this message] -j = : just output all of the programs and citations then exit (default = NO) -v = : verbose output? (default = NO) -d = : directory created by initPipeline (REQUIRED)

[options]: [pipeline_opts] [misc_opts]

[pipeline_opts]: options that affect the pipeline execution Pipeline consists of the following steps: Preprocess, Assemble, FindORFS, MapReads, Abundance, Annotate, FunctionalAnnotation, Scaffold, Propagate, Classify, Postprocess Each of these steps can be referred to by the following options: -f = : force this step to be run (default = NONE) -s = : start at this step in the pipeline (default = Preprocess) -e = : end at this step in the pipeline (default = Postprocess) -n = : step to skip in pipeline (default=NONE)

For each step you can fine-tune the execution as follows [Preprocess] -t = : filter input reads? (default = NO) -q = : produce FastQC quality report for reads with quality information (fastq or sff)? (default = NO) [Assemble] -a = : genome assembler to use (default = SOAPdenovo) -k = : k-mer size to be used for assembly (default = 31) -o = >: min overlap length [MapReads] -m = : read mapper to use? (default = bowtie) -i = : save bowtie (i)ndex? (default = NO) -b = : create library specific per bp coverage of assembled contigs (default = NO) [FindORFS] -g = : gene caller to use (default=FragGeneScan) -l = : min contig length to use for ORF call (default = 300) -x = >: min contig coverage to use for ORF call (default = 3X) [Annotate] -c = : classifier to use for annotation (default = FCP) -u = : annotate unassembled reads? (default = NO) [Classify] -z = : taxonomic level to categorize at (default = class)

[misc_opts]: Miscellaneous options -B = : blast DBs not available (default = NO) -r = : retain the AMOS bank? (default = NO) -p = : number of threads to use (be greedy!) (default=1) -4 = : 454 data? (default = NO) -L = : generate local Krona plots. Local Krona plots can only be viewed on the machine they are generated on but will work on a system with no internet connection (default = NO) [zzg@fat-node-1 ~]$

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub.

skoren commented 10 years ago

The binary is self contained so it does not require any python to run. Python and its dependencies is unpacked when the runPipeline binary runs. The only dependencies it has is on perl 5.8.8+ and java 1.6+. So you should be fine to run on your system.

Sergey

On Nov 13, 2013, at 9:06 PM, metamoszhang notifications@github.com wrote:

Hi Sergey ,

Many thanks for your response. I got it. Now, I want to know which python version has been included binary release. My python is 2.7.3. I think, my python should no problem for running metAMOS.

  • Zhigang Zhang

Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-65199268 Fax: 86-871-65199318

From: Sergey Koren Date: 2013-11-14 09:31 To: treangen/metAMOS CC: metamoszhang Subject: Re: [metAMOS] Could not import psutil, disabling - runPipeline (#90) Hi,

Psutil should be included in the binary release but this could be a system-specific issue causing the warning. Installing psutils on your system python will not change the version of python/packages being used by the binary release since it is fully self-contained. However, psutil is only used to estimate available CPU/memory so as long as you specify the number of cpus to use for the pipeline and have at least 32GB of ram available, you should be able to use metAMOS without issues.

Sergey

On Nov 12, 2013, at 10:24 PM, metamoszhang notifications@github.com wrote:

Hi,

I am using a frozen binary MetAMOS with all BLAST DBs. But when I test runPipeline command, an error "Could not import psutil, disabling." was reported. Now, I used Python 2.7.3 and tried to install psutil-0.5.1, psutil 0.6.1, and psutil-1.1.3. But the error was always reported. Please help me resolve it.

My python: [zzg@fat-node-1 ~]$ python Python 2.7.3 (default, Jun 6 2013, 17:48:01) [GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

psutil install:

[zzg@fat-node-1 psutil-0.5.1]$ python setup.py install running install running bdist_egg running egg_info writing psutil.egg-info/PKG-INFO writing top-level names to psutil.egg-info/top_level.txt writing dependency_links to psutil.egg-info/dependency_links.txt reading manifest file 'psutil.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '.html' under directory 'docs' warning: no files found matching '.dia' under directory 'docs' warning: no files found matching '*.png' under directory 'docs' writing manifest file 'psutil.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/psutil copying psutil/psposix.py -> build/lib.linux-x8664-2.7/psutil copying psutil/common.py -> build/lib.linux-x8664-2.7/psutil copying psutil/init.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psmswindows.py -> build/lib.linux-x8664-2.7/psutil copying psutil/pslinux.py -> build/lib.linux-x8664-2.7/psutil copying psutil/error.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psosx.py -> build/lib.linux-x8664-2.7/psutil copying psutil/psbsd.py -> build/lib.linux-x8664-2.7/psutil copying psutil/compat.py -> build/lib.linux-x8664-2.7/psutil running build_ext building 'psutillinux' extension creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/psutil gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutillinux.c -o build/temp.linux-x86_64-2.7/psutil/psutillinux.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutillinux.o -o build/lib.linux-x86_64-2.7/psutillinux.so building 'psutilposix' extension gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutilposix.c -o build/temp.linux-x86_64-2.7/psutil/psutilposix.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutilposix.o -o build/lib.linux-x86_64-2.7/psutilposix.so creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutilposix.so -> build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutillinux.so -> build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psposix.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/common.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/init.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psmswindows.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/pslinux.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/error.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psosx.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psbsd.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/compat.py -> build/bdist.linux-x8664/egg/psutil byte-compiling build/bdist.linux-x86_64/egg/psutil/psposix.py to _psposix.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/common.py to _common.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psmswindows.py to _psmswindows.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/pslinux.py to _pslinux.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/error.py to error.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psosx.py to _psosx.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/psbsd.py to _psbsd.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/compat.py to _compat.pyc creating stub loader for _psutillinux.so creating stub loader for psutilposix.so byte-compiling build/bdist.linux-x86_64/egg/psutillinux.py to psutillinux.pyc byte-compiling build/bdist.linux-x86_64/egg/psutilposix.py to psutilposix.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt zip_safe flag not set; analyzing archive contents... creating dist creating 'dist/psutil-0.5.1-py2.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing psutil-0.5.1-py2.7-linux-x86_64.egg Copying psutil-0.5.1-py2.7-linux-x86_64.egg to /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages Removing psutil 0.6.1 from easy-install.pth file Adding psutil 0.5.1 to easy-install.pth file

Installed /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages/psutil-0.5.1-py2.7-linux-x86_64.egg Processing dependencies for psutil==0.5.1 Finished processing dependencies for psutil==0.5.1

runPipeline test:

[zzg@fat-node-1 ~]$ runPipeline -h Could not import psutil, disabling. Could not import psutil, disabling. usage: runPipeline [options] -d projectdir -h = : print help [this message] -j = : just output all of the programs and citations then exit (default = NO) -v = : verbose output? (default = NO) -d = : directory created by initPipeline (REQUIRED)

[options]: [pipeline_opts] [misc_opts]

[pipeline_opts]: options that affect the pipeline execution Pipeline consists of the following steps: Preprocess, Assemble, FindORFS, MapReads, Abundance, Annotate, FunctionalAnnotation, Scaffold, Propagate, Classify, Postprocess Each of these steps can be referred to by the following options: -f = : force this step to be run (default = NONE) -s = : start at this step in the pipeline (default = Preprocess) -e = : end at this step in the pipeline (default = Postprocess) -n = : step to skip in pipeline (default=NONE)

For each step you can fine-tune the execution as follows [Preprocess] -t = : filter input reads? (default = NO) -q = : produce FastQC quality report for reads with quality information (fastq or sff)? (default = NO) [Assemble] -a = : genome assembler to use (default = SOAPdenovo) -k = : k-mer size to be used for assembly (default = 31) -o = >: min overlap length [MapReads] -m = : read mapper to use? (default = bowtie) -i = : save bowtie (i)ndex? (default = NO) -b = : create library specific per bp coverage of assembled contigs (default = NO) [FindORFS] -g = : gene caller to use (default=FragGeneScan) -l = : min contig length to use for ORF call (default = 300) -x = >: min contig coverage to use for ORF call (default = 3X) [Annotate] -c = : classifier to use for annotation (default = FCP) -u = : annotate unassembled reads? (default = NO) [Classify] -z = : taxonomic level to categorize at (default = class)

[misc_opts]: Miscellaneous options -B = : blast DBs not available (default = NO) -r = : retain the AMOS bank? (default = NO) -p = : number of threads to use (be greedy!) (default=1) -4 = : 454 data? (default = NO) -L = : generate local Krona plots. Local Krona plots can only be viewed on the machine they are generated on but will work on a system with no internet connection (default = NO) [zzg@fat-node-1 ~]$

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub. — Reply to this email directly or view it on GitHub.

metamoszhang commented 10 years ago

Ok. Thanks.

Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-65199268 Fax: 86-871-65199318

From: Sergey Koren Date: 2013-11-14 10:45 To: treangen/metAMOS CC: metamoszhang Subject: Re: [metAMOS] Could not import psutil, disabling - runPipeline (#90) The binary is self contained so it does not require any python to run. Python and its dependencies is unpacked when the runPipeline binary runs. The only dependencies it has is on perl 5.8.8+ and java 1.6+. So you should be fine to run on your system.

Sergey

On Nov 13, 2013, at 9:06 PM, metamoszhang notifications@github.com wrote:

Hi Sergey ,

Many thanks for your response. I got it. Now, I want to know which python version has been included binary release. My python is 2.7.3. I think, my python should no problem for running metAMOS.

  • Zhigang Zhang

Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-65199268 Fax: 86-871-65199318

From: Sergey Koren Date: 2013-11-14 09:31 To: treangen/metAMOS CC: metamoszhang Subject: Re: [metAMOS] Could not import psutil, disabling - runPipeline (#90) Hi,

Psutil should be included in the binary release but this could be a system-specific issue causing the warning. Installing psutils on your system python will not change the version of python/packages being used by the binary release since it is fully self-contained. However, psutil is only used to estimate available CPU/memory so as long as you specify the number of cpus to use for the pipeline and have at least 32GB of ram available, you should be able to use metAMOS without issues.

Sergey

On Nov 12, 2013, at 10:24 PM, metamoszhang notifications@github.com wrote:

Hi,

I am using a frozen binary MetAMOS with all BLAST DBs. But when I test runPipeline command, an error "Could not import psutil, disabling." was reported. Now, I used Python 2.7.3 and tried to install psutil-0.5.1, psutil 0.6.1, and psutil-1.1.3. But the error was always reported. Please help me resolve it.

My python: [zzg@fat-node-1 ~]$ python Python 2.7.3 (default, Jun 6 2013, 17:48:01) [GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

psutil install:

[zzg@fat-node-1 psutil-0.5.1]$ python setup.py install running install running bdist_egg running egg_info writing psutil.egg-info/PKG-INFO writing top-level names to psutil.egg-info/top_level.txt writing dependency_links to psutil.egg-info/dependency_links.txt reading manifest file 'psutil.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '.html' under directory 'docs' warning: no files found matching '.dia' under directory 'docs' warning: no files found matching '*.png' under directory 'docs' writing manifest file 'psutil.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/psutil copying psutil/psposix.py -> build/lib.linux-x8664-2.7/psutil copying psutil/common.py -> build/lib.linux-x8664-2.7/psutil copying psutil/init.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psmswindows.py -> build/lib.linux-x8664-2.7/psutil copying psutil/pslinux.py -> build/lib.linux-x8664-2.7/psutil copying psutil/error.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psosx.py -> build/lib.linux-x8664-2.7/psutil copying psutil/psbsd.py -> build/lib.linux-x8664-2.7/psutil copying psutil/compat.py -> build/lib.linux-x8664-2.7/psutil running build_ext building 'psutillinux' extension creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/psutil gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutillinux.c -o build/temp.linux-x86_64-2.7/psutil/psutillinux.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutillinux.o -o build/lib.linux-x86_64-2.7/psutillinux.so building 'psutilposix' extension gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutilposix.c -o build/temp.linux-x86_64-2.7/psutil/psutilposix.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutilposix.o -o build/lib.linux-x86_64-2.7/psutilposix.so creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutilposix.so -> build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutillinux.so -> build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psposix.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/common.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/init.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psmswindows.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/pslinux.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/error.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psosx.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psbsd.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/compat.py -> build/bdist.linux-x8664/egg/psutil byte-compiling build/bdist.linux-x86_64/egg/psutil/psposix.py to _psposix.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/common.py to _common.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psmswindows.py to _psmswindows.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/pslinux.py to _pslinux.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/error.py to error.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psosx.py to _psosx.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/psbsd.py to _psbsd.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/compat.py to _compat.pyc creating stub loader for _psutillinux.so creating stub loader for psutilposix.so byte-compiling build/bdist.linux-x86_64/egg/psutillinux.py to psutillinux.pyc byte-compiling build/bdist.linux-x86_64/egg/psutilposix.py to psutilposix.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt zip_safe flag not set; analyzing archive contents... creating dist creating 'dist/psutil-0.5.1-py2.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing psutil-0.5.1-py2.7-linux-x86_64.egg Copying psutil-0.5.1-py2.7-linux-x86_64.egg to /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages Removing psutil 0.6.1 from easy-install.pth file Adding psutil 0.5.1 to easy-install.pth file

Installed /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages/psutil-0.5.1-py2.7-linux-x86_64.egg Processing dependencies for psutil==0.5.1 Finished processing dependencies for psutil==0.5.1

runPipeline test:

[zzg@fat-node-1 ~]$ runPipeline -h Could not import psutil, disabling. Could not import psutil, disabling. usage: runPipeline [options] -d projectdir -h = : print help [this message] -j = : just output all of the programs and citations then exit (default = NO) -v = : verbose output? (default = NO) -d = : directory created by initPipeline (REQUIRED)

[options]: [pipeline_opts] [misc_opts]

[pipeline_opts]: options that affect the pipeline execution Pipeline consists of the following steps: Preprocess, Assemble, FindORFS, MapReads, Abundance, Annotate, FunctionalAnnotation, Scaffold, Propagate, Classify, Postprocess Each of these steps can be referred to by the following options: -f = : force this step to be run (default = NONE) -s = : start at this step in the pipeline (default = Preprocess) -e = : end at this step in the pipeline (default = Postprocess) -n = : step to skip in pipeline (default=NONE)

For each step you can fine-tune the execution as follows [Preprocess] -t = : filter input reads? (default = NO) -q = : produce FastQC quality report for reads with quality information (fastq or sff)? (default = NO) [Assemble] -a = : genome assembler to use (default = SOAPdenovo) -k = : k-mer size to be used for assembly (default = 31) -o = >: min overlap length [MapReads] -m = : read mapper to use? (default = bowtie) -i = : save bowtie (i)ndex? (default = NO) -b = : create library specific per bp coverage of assembled contigs (default = NO) [FindORFS] -g = : gene caller to use (default=FragGeneScan) -l = : min contig length to use for ORF call (default = 300) -x = >: min contig coverage to use for ORF call (default = 3X) [Annotate] -c = : classifier to use for annotation (default = FCP) -u = : annotate unassembled reads? (default = NO) [Classify] -z = : taxonomic level to categorize at (default = class)

[misc_opts]: Miscellaneous options -B = : blast DBs not available (default = NO) -r = : retain the AMOS bank? (default = NO) -p = : number of threads to use (be greedy!) (default=1) -4 = : 454 data? (default = NO) -L = : generate local Krona plots. Local Krona plots can only be viewed on the machine they are generated on but will work on a system with no internet connection (default = NO) [zzg@fat-node-1 ~]$

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub. — Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub.

metamoszhang commented 10 years ago

Hi Sergey Koren,

I am trying to test metAMOS (binary release) using my filtered high-quality dataset (167Gb fastq files) including three shortpaired reads libs. Please see below: Lib1: AD01_QC.fastq.1, AD01_QC.fastq.2; Lib2: AD02_QC.fastq.1, AD02_QC.fastq.2; Lib3: AD03_QC.fastq.1, AD03_QC.fastq.2. They were placed within the directory: /lustre/user/zzg/Project_ZhangZG/Paired.

List part of my data:

[zzg@manger-server in]$ head AD01_QC.fastq.1 @HWI-964:267:D23ARACXX:6:1101:1144:2168 1:N:0:GGCTAC AATAATGGTCATCAGTGAAGCCATAAGCCTCACGCTTCATCACCTTAATTTTTGTAAAAATGAATGGAGAAAGCTGACAATCCATTATCT + FBH@>EGGFDHHC9GCEF?EC>@CFHG@GGHIIGGGFHGGH @FHGH4=FHFGHIDHGDE>?EHFFCE?B@ACCCCCCCCCCCC:@CCD@C @HWI-964:267:D23ARACXX:6:1101:1207:2228 1:N:0:GGCTAC TGCCTCCGCAGGATGACAAGACGCAGAAGAAAAATCCTGCGGTTAAGGCTCAGCCTACGCTACAAATAGCGGACGACACCATCCCCGATT + FHGBHGCFHACF?EEGHII;G<F<HGG>@@FH@HIIGIBG'5:;B7;B>CCECC@CCCC <=CCAC>:ABB0<BB5&5?8@CA<<@.5@ @HWI-964:267:D23ARACXX:6:1101:1431:2192 1:N:0:GGCTAC TGACGATGTGGAGGGCGATGAGGCTTTGGCACATTTCTCGATGAAGGTGCATCAAGACGGATTCCCGAAAGAAGAGTATCCGCAGATTAA [zzg@manger-server in]$ head AD01_QC.fastq.2 @HWI-964:267:D23ARACXX:6:1101:1144:2168 2:N:0:GGCTAC TGGTCACAACCACAAACACGGCATTTGATGCGTTGAACGCGCATGTCGATGTAGACCGCCTTGCGACCGATCGGAACACAACGGAAACGA + ?FFAFBHIIIIIIIIIIIIIIIIFIBGHBHGIFGIGIIIIBEEF?CCA=@DBEDDCDDDDDDDDD<BDB@D

BDDDD?CBBDDBD@BCD@ @HWI-964:267:D23ARACXX:6:1101:1207:2228 2:N:0:GGCTAC GTGCATATCGGCAAACGAGAATTTCTCCTTGCCCTGCGTGGCAACATTCTCTGACTCCTTCTTGCGGAAGAAAGCCTCGCGCTCGCGACG + =FHGIIGHIIIBDHIIBDGG0BDGGDFH@FGCG CHDGEHA<CCEEECDDDFFAEEEEEDCDDDDDDB?>BCCC(<CC@DDD<@@DDBB @HWI-964:267:D23ARACXX:6:1101:1431:2192 2:N:0:GGCTAC TTGCCCATCGGCTCGTTCTCCGGCGTAATAGCCCAGAAGCGGATACCGGCAAATTCATATGCCTTCAGGTACTTCACAAAATAGCGTGCA

Now, I used PBS script for running metAMOS in my supercompuper system:

!/bin/sh

PBS -N metaAMOS

PBS -q huge

PBS -l nodes=1:ppn=6

PBS -o /lustre/user/zzg/output/metAMOS-rumen.out

PBS -e /lustre/user/zzg/output/metAMOS-rumen.err

cd $PBS_O_WORKDIR module load metAMOS

cd /lustre/user/zzg/Project_ZhangZG/Paired

initPipeline -1 AD01_QC.fastq.1,AD02_QC.fastq.1,AD03_QC.fastq.1 -2 AD01_QC.fastq.2,AD02_QC.fastq.2,AD03_QC.fastq.2 -d Cow_metAMOS_test -i 300:500,300:500,300:500 runPipeline -a SOAPdenovo,metavelvet -k 31 -p 6 -i YES -c phylosift -d Cow_metAMOS_test | tee metAMOS.logfile

I found initPipeline can sussessfully creat Project dir /lustre/user/zzg/Project_ZhangZG/Paired/Cow_metAMOS_test and can read my data to Preprocess step. You can see the result below:

[zzg@manger-server in]$ pwd /lustre/user/zzg/Project_ZhangZG/Paired/Cow_metAMOS_test/Preprocess/in [zzg@manger-server in]$ ll total 174896600 -rw-r--r-- 1 zzg ship 20225774056 Nov 16 23:16 AD01_QC.fastq.1 -rw-r--r-- 1 zzg ship 20225774056 Nov 16 23:17 AD01_QC.fastq.2 -rw-r--r-- 1 zzg ship 26015389018 Nov 16 23:19 AD02_QC.fastq.1 -rw-r--r-- 1 zzg ship 26015389018 Nov 16 23:20 AD02_QC.fastq.2 -rw-r--r-- 1 zzg ship 43305841961 Nov 16 23:23 AD03_QC.fastq.1 -rw-r--r-- 1 zzg ship 43305841961 Nov 16 23:26 AD03_QC.fastq.2 [zzg@manger-server in]$

However, I found runPipeline would be stopped when running Assemble step becuase I did not find any input and output in Assemble directory. Here, I did not know what will result in its stop. One possibility may be due to lack of genome assembler (whether its installation was performed by user or not ) in binary release. Or please tell me other possible reasons. Many thanks.

Zhigang Zhang

On Thu, Nov 14, 2013 at 10:45 AM, Sergey Koren notifications@github.comwrote:

The binary is self contained so it does not require any python to run. Python and its dependencies is unpacked when the runPipeline binary runs. The only dependencies it has is on perl 5.8.8+ and java 1.6+. So you should be fine to run on your system.

Sergey

On Nov 13, 2013, at 9:06 PM, metamoszhang notifications@github.com wrote:

Hi Sergey ,

Many thanks for your response. I got it. Now, I want to know which python version has been included binary release. My python is 2.7.3. I think, my python should no problem for running metAMOS.

  • Zhigang Zhang

Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-65199268 Fax: 86-871-65199318

From: Sergey Koren Date: 2013-11-14 09:31 To: treangen/metAMOS CC: metamoszhang Subject: Re: [metAMOS] Could not import psutil, disabling - runPipeline (#90) Hi,

Psutil should be included in the binary release but this could be a system-specific issue causing the warning. Installing psutils on your system python will not change the version of python/packages being used by the binary release since it is fully self-contained. However, psutil is only used to estimate available CPU/memory so as long as you specify the number of cpus to use for the pipeline and have at least 32GB of ram available, you should be able to use metAMOS without issues.

Sergey

On Nov 12, 2013, at 10:24 PM, metamoszhang notifications@github.com wrote:

Hi,

I am using a frozen binary MetAMOS with all BLAST DBs. But when I test runPipeline command, an error "Could not import psutil, disabling." was reported. Now, I used Python 2.7.3 and tried to install psutil-0.5.1, psutil 0.6.1, and psutil-1.1.3. But the error was always reported. Please help me resolve it.

My python: [zzg@fat-node-1 ~]$ python Python 2.7.3 (default, Jun 6 2013, 17:48:01) [GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

psutil install:

[zzg@fat-node-1 psutil-0.5.1]$ python setup.py install running install running bdist_egg running egg_info writing psutil.egg-info/PKG-INFO writing top-level names to psutil.egg-info/top_level.txt writing dependency_links to psutil.egg-info/dependency_links.txt reading manifest file 'psutil.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '.html' under directory 'docs' warning: no files found matching '.dia' under directory 'docs' warning: no files found matching '*.png' under directory 'docs' writing manifest file 'psutil.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/psutil copying psutil/psposix.py -> build/lib.linux-x8664-2.7/psutil copying psutil/common.py -> build/lib.linux-x8664-2.7/psutil copying psutil/init.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psmswindows.py -> build/lib.linux-x8664-2.7/psutil copying psutil/pslinux.py -> build/lib.linux-x8664-2.7/psutil copying psutil/error.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psosx.py -> build/lib.linux-x8664-2.7/psutil copying psutil/psbsd.py -> build/lib.linux-x8664-2.7/psutil copying psutil/compat.py -> build/lib.linux-x8664-2.7/psutil running build_ext building 'psutillinux' extension creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/psutil gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutillinux.c -o build/temp.linux-x86_64-2.7/psutil/psutillinux.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutillinux.o -o build/lib.linux-x86_64-2.7/psutillinux.so building 'psutilposix' extension gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutilposix.c -o build/temp.linux-x86_64-2.7/psutil/psutilposix.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutilposix.o -o build/lib.linux-x86_64-2.7/psutilposix.so creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutilposix.so -> build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutillinux.so -> build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psposix.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/common.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/init.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psmswindows.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/pslinux.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/error.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psosx.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psbsd.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/compat.py -> build/bdist.linux-x8664/egg/psutil byte-compiling build/bdist.linux-x86_64/egg/psutil/psposix.py to _psposix.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/common.py to _common.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psmswindows.py to _psmswindows.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/pslinux.py to _pslinux.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/error.py to error.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psosx.py to _psosx.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/psbsd.py to _psbsd.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/compat.py to _compat.pyc creating stub loader for _psutillinux.so creating stub loader for psutilposix.so byte-compiling build/bdist.linux-x86_64/egg/psutillinux.py to psutillinux.pyc byte-compiling build/bdist.linux-x86_64/egg/psutilposix.py to psutilposix.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt zip_safe flag not set; analyzing archive contents... creating dist creating 'dist/psutil-0.5.1-py2.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing psutil-0.5.1-py2.7-linux-x86_64.egg Copying psutil-0.5.1-py2.7-linux-x86_64.egg to /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages Removing psutil 0.6.1 from easy-install.pth file Adding psutil 0.5.1 to easy-install.pth file

Installed /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages/psutil-0.5.1-py2.7-linux-x86_64.egg

Processing dependencies for psutil==0.5.1 Finished processing dependencies for psutil==0.5.1

runPipeline test:

[zzg@fat-node-1 ~]$ runPipeline -h Could not import psutil, disabling. Could not import psutil, disabling. usage: runPipeline [options] -d projectdir -h = : print help [this message] -j = : just output all of the programs and citations then exit (default = NO) -v = : verbose output? (default = NO) -d = : directory created by initPipeline (REQUIRED)

[options]: [pipeline_opts] [misc_opts]

[pipeline_opts]: options that affect the pipeline execution Pipeline consists of the following steps: Preprocess, Assemble, FindORFS, MapReads, Abundance, Annotate, FunctionalAnnotation, Scaffold, Propagate, Classify, Postprocess Each of these steps can be referred to by the following options: -f = : force this step to be run (default = NONE) -s = : start at this step in the pipeline (default = Preprocess) -e = : end at this step in the pipeline (default = Postprocess) -n = : step to skip in pipeline (default=NONE)

For each step you can fine-tune the execution as follows [Preprocess] -t = : filter input reads? (default = NO) -q = : produce FastQC quality report for reads with quality information (fastq or sff)? (default = NO) [Assemble] -a = : genome assembler to use (default = SOAPdenovo) -k = : k-mer size to be used for assembly (default = 31) -o = >: min overlap length [MapReads] -m = : read mapper to use? (default = bowtie) -i = : save bowtie (i)ndex? (default = NO) -b = : create library specific per bp coverage of assembled contigs (default = NO) [FindORFS] -g = : gene caller to use (default=FragGeneScan) -l = : min contig length to use for ORF call (default = 300) -x = >: min contig coverage to use for ORF call (default = 3X) [Annotate] -c = : classifier to use for annotation (default = FCP) -u = : annotate unassembled reads? (default = NO) [Classify] -z = : taxonomic level to categorize at (default = class)

[misc_opts]: Miscellaneous options -B = : blast DBs not available (default = NO) -r = : retain the AMOS bank? (default = NO) -p = : number of threads to use (be greedy!) (default=1) -4 = : 454 data? (default = NO) -L = : generate local Krona plots. Local Krona plots can only be viewed on the machine they are generated on but will work on a system with no internet connection (default = NO) [zzg@fat-node-1 ~]$

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub. — Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/treangen/metAMOS/issues/90#issuecomment-28455215 .

Zhigang Zhang Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-5199268 Fax: 86-871-5199318

skoren commented 10 years ago

Hi,

The frozen binary should include both soapdenovo and metavelvet in the distribution. metAMOS will also continue as long as one of the assemblers was successful. If all assemblies failed, then metAMOS will stop since it cannot continue. The output from runPipeline should indicate the exact error that was encountered which caused the pipeline to stop. Could you send that output along with the assemble step output (found in /Logs/ASSEMBLE.log).

Sergey On Nov 16, 2013, at 11:08 AM, metamoszhang notifications@github.com wrote:

Hi Sergey Koren,

I am trying to test metAMOS (binary release) using my filtered high-quality dataset (167Gb fastq files) including three shortpaired reads libs. Please see below: Lib1: AD01_QC.fastq.1, AD01_QC.fastq.2; Lib2: AD02_QC.fastq.1, AD02_QC.fastq.2; Lib3: AD03_QC.fastq.1, AD03_QC.fastq.2. They were placed within the directory: /lustre/user/zzg/Project_ZhangZG/Paired.

List part of my data:

[zzg@manger-server in]$ head AD01_QC.fastq.1 @HWI-964:267:D23ARACXX:6:1101:1144:2168 1:N:0:GGCTAC AATAATGGTCATCAGTGAAGCCATAAGCCTCACGCTTCATCACCTTAATTTTTGTAAAAATGAATGGAGAAAGCTGACAATCCATTATCT

  • FBH@>EGGFDHHC9GCEF?EC>@CFHG@GGHIIGGGFHGGH @FHGH4=FHFGHIDHGDE>?EHFFCE?B@ACCCCCCCCCCCC:@CCD@C @HWI-964:267:D23ARACXX:6:1101:1207:2228 1:N:0:GGCTAC TGCCTCCGCAGGATGACAAGACGCAGAAGAAAAATCCTGCGGTTAAGGCTCAGCCTACGCTACAAATAGCGGACGACACCATCCCCGATT

  • FHGBHGCFHACF?EEGHII;G<F<HGG>@@FH@HIIGIBG'5:;B7;B>CCECC@CCCC <=CCAC>:ABB0<BB5&5?8@CA<<@.5@ @HWI-964:267:D23ARACXX:6:1101:1431:2192 1:N:0:GGCTAC TGACGATGTGGAGGGCGATGAGGCTTTGGCACATTTCTCGATGAAGGTGCATCAAGACGGATTCCCGAAAGAAGAGTATCCGCAGATTAA [zzg@manger-server in]$ head AD01_QC.fastq.2 @HWI-964:267:D23ARACXX:6:1101:1144:2168 2:N:0:GGCTAC TGGTCACAACCACAAACACGGCATTTGATGCGTTGAACGCGCATGTCGATGTAGACCGCCTTGCGACCGATCGGAACACAACGGAAACGA

  • ?FFAFBHIIIIIIIIIIIIIIIIFIBGHBHGIFGIGIIIIBEEF?CCA=@DBEDDCDDDDDDDDD<BDB@D

    BDDDD?CBBDDBD@BCD@ @HWI-964:267:D23ARACXX:6:1101:1207:2228 2:N:0:GGCTAC GTGCATATCGGCAAACGAGAATTTCTCCTTGCCCTGCGTGGCAACATTCTCTGACTCCTTCTTGCGGAAGAAAGCCTCGCGCTCGCGACG

  • =FHGIIGHIIIBDHIIBDGG0BDGGDFH@FGCG CHDGEHA<CCEEECDDDFFAEEEEEDCDDDDDDB?>BCCC(<CC@DDD<@@DDBB @HWI-964:267:D23ARACXX:6:1101:1431:2192 2:N:0:GGCTAC TTGCCCATCGGCTCGTTCTCCGGCGTAATAGCCCAGAAGCGGATACCGGCAAATTCATATGCCTTCAGGTACTTCACAAAATAGCGTGCA

Now, I used PBS script for running metAMOS in my supercompuper system:

!/bin/sh

PBS -N metaAMOS

PBS -q huge

PBS -l nodes=1:ppn=6

PBS -o /lustre/user/zzg/output/metAMOS-rumen.out

PBS -e /lustre/user/zzg/output/metAMOS-rumen.err

cd $PBS_O_WORKDIR module load metAMOS

cd /lustre/user/zzg/Project_ZhangZG/Paired

initPipeline -1 AD01_QC.fastq.1,AD02_QC.fastq.1,AD03_QC.fastq.1 -2 AD01_QC.fastq.2,AD02_QC.fastq.2,AD03_QC.fastq.2 -d Cow_metAMOS_test -i 300:500,300:500,300:500 runPipeline -a SOAPdenovo,metavelvet -k 31 -p 6 -i YES -c phylosift -d Cow_metAMOS_test | tee metAMOS.logfile

I found initPipeline can sussessfully creat Project dir /lustre/user/zzg/Project_ZhangZG/Paired/Cow_metAMOS_test and can read my data to Preprocess step. You can see the result below:

[zzg@manger-server in]$ pwd /lustre/user/zzg/Project_ZhangZG/Paired/Cow_metAMOS_test/Preprocess/in [zzg@manger-server in]$ ll total 174896600 -rw-r--r-- 1 zzg ship 20225774056 Nov 16 23:16 AD01_QC.fastq.1 -rw-r--r-- 1 zzg ship 20225774056 Nov 16 23:17 AD01_QC.fastq.2 -rw-r--r-- 1 zzg ship 26015389018 Nov 16 23:19 AD02_QC.fastq.1 -rw-r--r-- 1 zzg ship 26015389018 Nov 16 23:20 AD02_QC.fastq.2 -rw-r--r-- 1 zzg ship 43305841961 Nov 16 23:23 AD03_QC.fastq.1 -rw-r--r-- 1 zzg ship 43305841961 Nov 16 23:26 AD03_QC.fastq.2 [zzg@manger-server in]$

However, I found runPipeline would be stopped when running Assemble step becuase I did not find any input and output in Assemble directory. Here, I did not know what will result in its stop. One possibility may be due to lack of genome assembler (whether its installation was performed by user or not ) in binary release. Or please tell me other possible reasons. Many thanks.

Zhigang Zhang

On Thu, Nov 14, 2013 at 10:45 AM, Sergey Koren notifications@github.comwrote:

The binary is self contained so it does not require any python to run. Python and its dependencies is unpacked when the runPipeline binary runs. The only dependencies it has is on perl 5.8.8+ and java 1.6+. So you should be fine to run on your system.

Sergey

On Nov 13, 2013, at 9:06 PM, metamoszhang notifications@github.com wrote:

Hi Sergey ,

Many thanks for your response. I got it. Now, I want to know which python version has been included binary release. My python is 2.7.3. I think, my python should no problem for running metAMOS.

  • Zhigang Zhang

Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-65199268 Fax: 86-871-65199318

From: Sergey Koren Date: 2013-11-14 09:31 To: treangen/metAMOS CC: metamoszhang Subject: Re: [metAMOS] Could not import psutil, disabling - runPipeline (#90) Hi,

Psutil should be included in the binary release but this could be a system-specific issue causing the warning. Installing psutils on your system python will not change the version of python/packages being used by the binary release since it is fully self-contained. However, psutil is only used to estimate available CPU/memory so as long as you specify the number of cpus to use for the pipeline and have at least 32GB of ram available, you should be able to use metAMOS without issues.

Sergey

On Nov 12, 2013, at 10:24 PM, metamoszhang notifications@github.com wrote:

Hi,

I am using a frozen binary MetAMOS with all BLAST DBs. But when I test runPipeline command, an error "Could not import psutil, disabling." was reported. Now, I used Python 2.7.3 and tried to install psutil-0.5.1, psutil 0.6.1, and psutil-1.1.3. But the error was always reported. Please help me resolve it.

My python: [zzg@fat-node-1 ~]$ python Python 2.7.3 (default, Jun 6 2013, 17:48:01) [GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

psutil install:

[zzg@fat-node-1 psutil-0.5.1]$ python setup.py install running install running bdist_egg running egg_info writing psutil.egg-info/PKG-INFO writing top-level names to psutil.egg-info/top_level.txt writing dependency_links to psutil.egg-info/dependency_links.txt reading manifest file 'psutil.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '.html' under directory 'docs' warning: no files found matching '.dia' under directory 'docs' warning: no files found matching '*.png' under directory 'docs' writing manifest file 'psutil.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/psutil copying psutil/psposix.py -> build/lib.linux-x8664-2.7/psutil copying psutil/common.py -> build/lib.linux-x8664-2.7/psutil copying psutil/init.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psmswindows.py -> build/lib.linux-x8664-2.7/psutil copying psutil/pslinux.py -> build/lib.linux-x8664-2.7/psutil copying psutil/error.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psosx.py -> build/lib.linux-x8664-2.7/psutil copying psutil/psbsd.py -> build/lib.linux-x8664-2.7/psutil copying psutil/compat.py -> build/lib.linux-x8664-2.7/psutil running build_ext building 'psutillinux' extension creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/psutil gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutillinux.c -o build/temp.linux-x86_64-2.7/psutil/psutillinux.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutillinux.o -o build/lib.linux-x86_64-2.7/psutillinux.so building 'psutilposix' extension gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutilposix.c -o build/temp.linux-x86_64-2.7/psutil/psutilposix.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutilposix.o -o build/lib.linux-x86_64-2.7/psutilposix.so creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutilposix.so -> build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutillinux.so -> build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psposix.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/common.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/init.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psmswindows.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/pslinux.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/error.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psosx.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psbsd.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/compat.py -> build/bdist.linux-x8664/egg/psutil byte-compiling build/bdist.linux-x86_64/egg/psutil/psposix.py to _psposix.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/common.py to _common.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psmswindows.py to _psmswindows.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/pslinux.py to _pslinux.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/error.py to error.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psosx.py to _psosx.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/psbsd.py to _psbsd.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/compat.py to _compat.pyc creating stub loader for _psutillinux.so creating stub loader for psutilposix.so byte-compiling build/bdist.linux-x86_64/egg/psutillinux.py to psutillinux.pyc byte-compiling build/bdist.linux-x86_64/egg/psutilposix.py to psutilposix.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt zip_safe flag not set; analyzing archive contents... creating dist creating 'dist/psutil-0.5.1-py2.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing psutil-0.5.1-py2.7-linux-x86_64.egg Copying psutil-0.5.1-py2.7-linux-x86_64.egg to /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages Removing psutil 0.6.1 from easy-install.pth file Adding psutil 0.5.1 to easy-install.pth file

Installed /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages/psutil-0.5.1-py2.7-linux-x86_64.egg

Processing dependencies for psutil==0.5.1 Finished processing dependencies for psutil==0.5.1

runPipeline test:

[zzg@fat-node-1 ~]$ runPipeline -h Could not import psutil, disabling. Could not import psutil, disabling. usage: runPipeline [options] -d projectdir -h = : print help [this message] -j = : just output all of the programs and citations then exit (default = NO) -v = : verbose output? (default = NO) -d = : directory created by initPipeline (REQUIRED)

[options]: [pipeline_opts] [misc_opts]

[pipeline_opts]: options that affect the pipeline execution Pipeline consists of the following steps: Preprocess, Assemble, FindORFS, MapReads, Abundance, Annotate, FunctionalAnnotation, Scaffold, Propagate, Classify, Postprocess Each of these steps can be referred to by the following options: -f = : force this step to be run (default = NONE) -s = : start at this step in the pipeline (default = Preprocess) -e = : end at this step in the pipeline (default = Postprocess) -n = : step to skip in pipeline (default=NONE)

For each step you can fine-tune the execution as follows [Preprocess] -t = : filter input reads? (default = NO) -q = : produce FastQC quality report for reads with quality information (fastq or sff)? (default = NO) [Assemble] -a = : genome assembler to use (default = SOAPdenovo) -k = : k-mer size to be used for assembly (default = 31) -o = >: min overlap length [MapReads] -m = : read mapper to use? (default = bowtie) -i = : save bowtie (i)ndex? (default = NO) -b = : create library specific per bp coverage of assembled contigs (default = NO) [FindORFS] -g = : gene caller to use (default=FragGeneScan) -l = : min contig length to use for ORF call (default = 300) -x = >: min contig coverage to use for ORF call (default = 3X) [Annotate] -c = : classifier to use for annotation (default = FCP) -u = : annotate unassembled reads? (default = NO) [Classify] -z = : taxonomic level to categorize at (default = class)

[misc_opts]: Miscellaneous options -B = : blast DBs not available (default = NO) -r = : retain the AMOS bank? (default = NO) -p = : number of threads to use (be greedy!) (default=1) -4 = : 454 data? (default = NO) -L = : generate local Krona plots. Local Krona plots can only be viewed on the machine they are generated on but will work on a system with no internet connection (default = NO) [zzg@fat-node-1 ~]$

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub. — Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/treangen/metAMOS/issues/90#issuecomment-28455215 .

Zhigang Zhang Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-5199268 Fax: 86-871-5199318 — Reply to this email directly or view it on GitHub.

metamoszhang commented 10 years ago

Hi Sugery,

Many thanks for your responses. ASSEMBLE.log was not produced. I think, Assemle was still not started and only my data was inputted after run the below PBS script.

!/bin/sh

PBS -N metAMOS

PBS -q huge

PBS -l nodes=1:fat-node-2:ppn=4

PBS -o /lustre/user/zzg/output/metAMOS-rumen.out

PBS -e /lustre/user/zzg/output/metAMOS-rumen.err

cd $PBS_O_WORKDIR module load metAMOS cd /lustre/user/zzg/Project_ZhangZG/Paired initPipeline -1 AD01_QC.fastq.1,AD02_QC.fastq.1,AD03_QC.fastq.1 -2 AD01_QC.fastq.2,AD02_QC.fastq.2,AD03_QC.fastq.2 -d Cow_metAMOS_test -i 300:500,300:500,300:500 runPipeline -k 51 -p 6 -i YES -c phylosift -d Cow_metAMOS_test | tee metAMOS.logfile

[zzg@manger-server in]$ pwd /lustre/user/zzg/Project_ZhangZG/Paired/Cow_metAMOS_test/Preprocess/in [zzg@manger-server in]$ ll total 174896600 -rw-r--r-- 1 zzg ship 20225774056 Nov 16 23:16 AD01_QC.fastq.1 -rw-r--r-- 1 zzg ship 20225774056 Nov 16 23:17 AD01_QC.fastq.2 -rw-r--r-- 1 zzg ship 26015389018 Nov 16 23:19 AD02_QC.fastq.1 -rw-r--r-- 1 zzg ship 26015389018 Nov 16 23:20 AD02_QC.fastq.2 -rw-r--r-- 1 zzg ship 43305841961 Nov 16 23:23 AD03_QC.fastq.1 -rw-r--r-- 1 zzg ship 43305841961 Nov 16 23:26 AD03_QC.fastq.2 [zzg@manger-server in]$

The size of these three libs are 14.22, 18.29, and 31.23 Gb (90 bp length paired reads), respectively.

We can see metAMOS.logfile. I did not sure whether this issue is related to memory size or not because my memory size is limitted to 32 Gb.

[zzg@manger-server Paired]$ cat metAMOS.logfile Project dir /lustre/user/zzg/Project_ZhangZG/Paired/Cow_metAMOS_test successfully created! Use runPipeline.py to start Pipeline Could not import psutil, disabling. Could not import psutil, disabling. project dir does not exist! usage: runPipeline [options] -d projectdir -h = : print help [this message] -j = : just output all of the programs and citations then exit (default = NO) -v = : verbose output? (default = NO) -d = : directory created by initPipeline (REQUIRED)

[options]: [pipeline_opts] [misc_opts]

[pipeline_opts]: options that affect the pipeline execution Pipeline consists of the following steps: Preprocess, Assemble, FindORFS, MapReads, Abundance, Annotate, FunctionalAnnotation, Scaffold, Propagate, Classify, Postprocess Each of these steps can be referred to by the following options: -f = : force this step to be run (default = NONE) -s = : start at this step in the pipeline (default = Preprocess) -e = : end at this step in the pipeline (default = Postprocess) -n = : step to skip in pipeline (default=NONE)

For each step you can fine-tune the execution as follows [Preprocess] -t = : filter input reads? (default = NO) -q = : produce FastQC quality report for reads with quality information (fastq or sff)? (default = NO) [Assemble] -a = : genome assembler to use (default = SOAPdenovo) -k = : k-mer size to be used for assembly (default = 31) -o = >: min overlap length [MapReads] -m = : read mapper to use? (default = bowtie) -i = : save bowtie (i)ndex? (default = NO) -b = : create library specific per bp coverage of assembled contigs (default = NO) [FindORFS] -g = : gene caller to use (default=FragGeneScan) -l = : min contig length to use for ORF call (default = 300) -x = >: min contig coverage to use for ORF call (default = 3X) [Annotate] -c = : classifier to use for annotation (default = FCP) -u = : annotate unassembled reads? (default = NO) [Classify] -z = : taxonomic level to categorize at (default = class)

[misc_opts]: Miscellaneous options -B = : blast DBs not available (default = NO) -r = : retain the AMOS bank? (default = NO) -p = : number of threads to use (be greedy!) (default=1) -4 = : 454 data? (default = NO) -L = : generate local Krona plots. Local Krona plots can only be viewed on the machine they are generated on but will work on a system with no internet connection (default = NO) [zzg@manger-server Paired]$

Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-65199268 Fax: 86-871-65199318

From: Sergey Koren Date: 2013-11-20 06:21 To: treangen/metAMOS CC: metamoszhang Subject: Re: [metAMOS] Could not import psutil, disabling - runPipeline (#90) Hi,

The frozen binary should include both soapdenovo and metavelvet in the distribution. metAMOS will also continue as long as one of the assemblers was successful. If all assemblies failed, then metAMOS will stop since it cannot continue. The output from runPipeline should indicate the exact error that was encountered which caused the pipeline to stop. Could you send that output along with the assemble step output (found in /Logs/ASSEMBLE.log).

Sergey On Nov 16, 2013, at 11:08 AM, metamoszhang notifications@github.com wrote:

Hi Sergey Koren,

I am trying to test metAMOS (binary release) using my filtered high-quality dataset (167Gb fastq files) including three shortpaired reads libs. Please see below: Lib1: AD01_QC.fastq.1, AD01_QC.fastq.2; Lib2: AD02_QC.fastq.1, AD02_QC.fastq.2; Lib3: AD03_QC.fastq.1, AD03_QC.fastq.2. They were placed within the directory: /lustre/user/zzg/Project_ZhangZG/Paired.

List part of my data:

[zzg@manger-server in]$ head AD01_QC.fastq.1 @HWI-964:267:D23ARACXX:6:1101:1144:2168 1:N:0:GGCTAC AATAATGGTCATCAGTGAAGCCATAAGCCTCACGCTTCATCACCTTAATTTTTGTAAAAATGAATGGAGAAAGCTGACAATCCATTATCT

  • FBH@>EGGFDHHC9GCEF?EC>@CFHG@GGHIIGGGFHGGH @FHGH4=FHFGHIDHGDE>?EHFFCE?B@ACCCCCCCCCCCC:@CCD@C @HWI-964:267:D23ARACXX:6:1101:1207:2228 1:N:0:GGCTAC TGCCTCCGCAGGATGACAAGACGCAGAAGAAAAATCCTGCGGTTAAGGCTCAGCCTACGCTACAAATAGCGGACGACACCATCCCCGATT

  • FHGBHGCFHACF?EEGHII;G<F<HGG>@@FH@HIIGIBG'5:;B7;B>CCECC@CCCC <=CCAC>:ABB0<BB5&5?8@CA<<@.5@ @HWI-964:267:D23ARACXX:6:1101:1431:2192 1:N:0:GGCTAC TGACGATGTGGAGGGCGATGAGGCTTTGGCACATTTCTCGATGAAGGTGCATCAAGACGGATTCCCGAAAGAAGAGTATCCGCAGATTAA [zzg@manger-server in]$ head AD01_QC.fastq.2 @HWI-964:267:D23ARACXX:6:1101:1144:2168 2:N:0:GGCTAC TGGTCACAACCACAAACACGGCATTTGATGCGTTGAACGCGCATGTCGATGTAGACCGCCTTGCGACCGATCGGAACACAACGGAAACGA

  • ?FFAFBHIIIIIIIIIIIIIIIIFIBGHBHGIFGIGIIIIBEEF?CCA=@DBEDDCDDDDDDDDD<BDB@D

    BDDDD?CBBDDBD@BCD@ @HWI-964:267:D23ARACXX:6:1101:1207:2228 2:N:0:GGCTAC GTGCATATCGGCAAACGAGAATTTCTCCTTGCCCTGCGTGGCAACATTCTCTGACTCCTTCTTGCGGAAGAAAGCCTCGCGCTCGCGACG

  • =FHGIIGHIIIBDHIIBDGG0BDGGDFH@FGCG CHDGEHA<CCEEECDDDFFAEEEEEDCDDDDDDB?>BCCC(<CC@DDD<@@DDBB @HWI-964:267:D23ARACXX:6:1101:1431:2192 2:N:0:GGCTAC TTGCCCATCGGCTCGTTCTCCGGCGTAATAGCCCAGAAGCGGATACCGGCAAATTCATATGCCTTCAGGTACTTCACAAAATAGCGTGCA

Now, I used PBS script for running metAMOS in my supercompuper system:

!/bin/sh

PBS -N metaAMOS

PBS -q huge

PBS -l nodes=1:ppn=6

PBS -o /lustre/user/zzg/output/metAMOS-rumen.out

PBS -e /lustre/user/zzg/output/metAMOS-rumen.err

cd $PBS_O_WORKDIR module load metAMOS

cd /lustre/user/zzg/Project_ZhangZG/Paired

initPipeline -1 AD01_QC.fastq.1,AD02_QC.fastq.1,AD03_QC.fastq.1 -2 AD01_QC.fastq.2,AD02_QC.fastq.2,AD03_QC.fastq.2 -d Cow_metAMOS_test -i 300:500,300:500,300:500 runPipeline -a SOAPdenovo,metavelvet -k 31 -p 6 -i YES -c phylosift -d Cow_metAMOS_test | tee metAMOS.logfile

I found initPipeline can sussessfully creat Project dir /lustre/user/zzg/Project_ZhangZG/Paired/Cow_metAMOS_test and can read my data to Preprocess step. You can see the result below:

[zzg@manger-server in]$ pwd /lustre/user/zzg/Project_ZhangZG/Paired/Cow_metAMOS_test/Preprocess/in [zzg@manger-server in]$ ll total 174896600 -rw-r--r-- 1 zzg ship 20225774056 Nov 16 23:16 AD01_QC.fastq.1 -rw-r--r-- 1 zzg ship 20225774056 Nov 16 23:17 AD01_QC.fastq.2 -rw-r--r-- 1 zzg ship 26015389018 Nov 16 23:19 AD02_QC.fastq.1 -rw-r--r-- 1 zzg ship 26015389018 Nov 16 23:20 AD02_QC.fastq.2 -rw-r--r-- 1 zzg ship 43305841961 Nov 16 23:23 AD03_QC.fastq.1 -rw-r--r-- 1 zzg ship 43305841961 Nov 16 23:26 AD03_QC.fastq.2 [zzg@manger-server in]$

However, I found runPipeline would be stopped when running Assemble step becuase I did not find any input and output in Assemble directory. Here, I did not know what will result in its stop. One possibility may be due to lack of genome assembler (whether its installation was performed by user or not ) in binary release. Or please tell me other possible reasons. Many thanks.

Zhigang Zhang

On Thu, Nov 14, 2013 at 10:45 AM, Sergey Koren notifications@github.comwrote:

The binary is self contained so it does not require any python to run. Python and its dependencies is unpacked when the runPipeline binary runs. The only dependencies it has is on perl 5.8.8+ and java 1.6+. So you should be fine to run on your system.

Sergey

On Nov 13, 2013, at 9:06 PM, metamoszhang notifications@github.com wrote:

Hi Sergey ,

Many thanks for your response. I got it. Now, I want to know which python version has been included binary release. My python is 2.7.3. I think, my python should no problem for running metAMOS.

  • Zhigang Zhang

Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-65199268 Fax: 86-871-65199318

From: Sergey Koren Date: 2013-11-14 09:31 To: treangen/metAMOS CC: metamoszhang Subject: Re: [metAMOS] Could not import psutil, disabling - runPipeline (#90) Hi,

Psutil should be included in the binary release but this could be a system-specific issue causing the warning. Installing psutils on your system python will not change the version of python/packages being used by the binary release since it is fully self-contained. However, psutil is only used to estimate available CPU/memory so as long as you specify the number of cpus to use for the pipeline and have at least 32GB of ram available, you should be able to use metAMOS without issues.

Sergey

On Nov 12, 2013, at 10:24 PM, metamoszhang notifications@github.com wrote:

Hi,

I am using a frozen binary MetAMOS with all BLAST DBs. But when I test runPipeline command, an error "Could not import psutil, disabling." was reported. Now, I used Python 2.7.3 and tried to install psutil-0.5.1, psutil 0.6.1, and psutil-1.1.3. But the error was always reported. Please help me resolve it.

My python: [zzg@fat-node-1 ~]$ python Python 2.7.3 (default, Jun 6 2013, 17:48:01) [GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

psutil install:

[zzg@fat-node-1 psutil-0.5.1]$ python setup.py install running install running bdist_egg running egg_info writing psutil.egg-info/PKG-INFO writing top-level names to psutil.egg-info/top_level.txt writing dependency_links to psutil.egg-info/dependency_links.txt reading manifest file 'psutil.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '.html' under directory 'docs' warning: no files found matching '.dia' under directory 'docs' warning: no files found matching '*.png' under directory 'docs' writing manifest file 'psutil.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/psutil copying psutil/psposix.py -> build/lib.linux-x8664-2.7/psutil copying psutil/common.py -> build/lib.linux-x8664-2.7/psutil copying psutil/init.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psmswindows.py -> build/lib.linux-x8664-2.7/psutil copying psutil/pslinux.py -> build/lib.linux-x8664-2.7/psutil copying psutil/error.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psosx.py -> build/lib.linux-x8664-2.7/psutil copying psutil/psbsd.py -> build/lib.linux-x8664-2.7/psutil copying psutil/compat.py -> build/lib.linux-x8664-2.7/psutil running build_ext building 'psutillinux' extension creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/psutil gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutillinux.c -o build/temp.linux-x86_64-2.7/psutil/psutillinux.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutillinux.o -o build/lib.linux-x86_64-2.7/psutillinux.so building 'psutilposix' extension gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutilposix.c -o build/temp.linux-x86_64-2.7/psutil/psutilposix.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutilposix.o -o build/lib.linux-x86_64-2.7/psutilposix.so creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutilposix.so -> build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutillinux.so -> build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psposix.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/common.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/init.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psmswindows.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/pslinux.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/error.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psosx.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psbsd.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/compat.py -> build/bdist.linux-x8664/egg/psutil byte-compiling build/bdist.linux-x86_64/egg/psutil/psposix.py to _psposix.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/common.py to _common.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psmswindows.py to _psmswindows.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/pslinux.py to _pslinux.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/error.py to error.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psosx.py to _psosx.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/psbsd.py to _psbsd.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/compat.py to _compat.pyc creating stub loader for _psutillinux.so creating stub loader for psutilposix.so byte-compiling build/bdist.linux-x86_64/egg/psutillinux.py to psutillinux.pyc byte-compiling build/bdist.linux-x86_64/egg/psutilposix.py to psutilposix.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt zip_safe flag not set; analyzing archive contents... creating dist creating 'dist/psutil-0.5.1-py2.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing psutil-0.5.1-py2.7-linux-x86_64.egg Copying psutil-0.5.1-py2.7-linux-x86_64.egg to /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages Removing psutil 0.6.1 from easy-install.pth file Adding psutil 0.5.1 to easy-install.pth file

Installed /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages/psutil-0.5.1-py2.7-linux-x86_64.egg

Processing dependencies for psutil==0.5.1 Finished processing dependencies for psutil==0.5.1

runPipeline test:

[zzg@fat-node-1 ~]$ runPipeline -h Could not import psutil, disabling. Could not import psutil, disabling. usage: runPipeline [options] -d projectdir -h = : print help [this message] -j = : just output all of the programs and citations then exit (default = NO) -v = : verbose output? (default = NO) -d = : directory created by initPipeline (REQUIRED)

[options]: [pipeline_opts] [misc_opts]

[pipeline_opts]: options that affect the pipeline execution Pipeline consists of the following steps: Preprocess, Assemble, FindORFS, MapReads, Abundance, Annotate, FunctionalAnnotation, Scaffold, Propagate, Classify, Postprocess Each of these steps can be referred to by the following options: -f = : force this step to be run (default = NONE) -s = : start at this step in the pipeline (default = Preprocess) -e = : end at this step in the pipeline (default = Postprocess) -n = : step to skip in pipeline (default=NONE)

For each step you can fine-tune the execution as follows [Preprocess] -t = : filter input reads? (default = NO) -q = : produce FastQC quality report for reads with quality information (fastq or sff)? (default = NO) [Assemble] -a = : genome assembler to use (default = SOAPdenovo) -k = : k-mer size to be used for assembly (default = 31) -o = >: min overlap length [MapReads] -m = : read mapper to use? (default = bowtie) -i = : save bowtie (i)ndex? (default = NO) -b = : create library specific per bp coverage of assembled contigs (default = NO) [FindORFS] -g = : gene caller to use (default=FragGeneScan) -l = : min contig length to use for ORF call (default = 300) -x = >: min contig coverage to use for ORF call (default = 3X) [Annotate] -c = : classifier to use for annotation (default = FCP) -u = : annotate unassembled reads? (default = NO) [Classify] -z = : taxonomic level to categorize at (default = class)

[misc_opts]: Miscellaneous options -B = : blast DBs not available (default = NO) -r = : retain the AMOS bank? (default = NO) -p = : number of threads to use (be greedy!) (default=1) -4 = : 454 data? (default = NO) -L = : generate local Krona plots. Local Krona plots can only be viewed on the machine they are generated on but will work on a system with no internet connection (default = NO) [zzg@fat-node-1 ~]$

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub. — Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/treangen/metAMOS/issues/90#issuecomment-28455215 .

Zhigang Zhang Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-5199268 Fax: 86-871-5199318 — Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub.

skoren commented 10 years ago

Hi,

Based on your log file, the pipeline never started runPipeline due to bad command-line parameters:

project dir does not exist! usage: runPipeline [options] -d projectdir This is because the -i option to runPipeline has no subsequent parameter. Your runPipeline command should be updated to: runPipeline -k 51 -p 6 -i -c phylosift -d Cow_metAMOS_test | tee metAMOS.logfile

Also, a warning, the dataset you are using is likely the largest dataset metAMOS has been run on. The assemble step will take a large amount of RAM, especially for metavelvet. I would definitely say 32GB Is not going to be enough. The preprocess and mapread steps will likely be slow. I would also recommend not using phylosift as a classifier as it can be slow on large datasets, instead I would recommend kraken.

Sergey

On Nov 19, 2013, at 10:48 PM, metamoszhang notifications@github.com wrote:

Hi Sugery,

Many thanks for your responses. ASSEMBLE.log was not produced. I think, Assemle was still not started and only my data was inputted after run the below PBS script.

!/bin/sh

PBS -N metAMOS

PBS -q huge

PBS -l nodes=1:fat-node-2:ppn=4

PBS -o /lustre/user/zzg/output/metAMOS-rumen.out

PBS -e /lustre/user/zzg/output/metAMOS-rumen.err

cd $PBS_O_WORKDIR module load metAMOS cd /lustre/user/zzg/Project_ZhangZG/Paired initPipeline -1 AD01_QC.fastq.1,AD02_QC.fastq.1,AD03_QC.fastq.1 -2 AD01_QC.fastq.2,AD02_QC.fastq.2,AD03_QC.fastq.2 -d Cow_metAMOS_test -i 300:500,300:500,300:500 runPipeline -k 51 -p 6 -i YES -c phylosift -d Cow_metAMOS_test | tee metAMOS.logfile

[zzg@manger-server in]$ pwd /lustre/user/zzg/Project_ZhangZG/Paired/Cow_metAMOS_test/Preprocess/in [zzg@manger-server in]$ ll total 174896600 -rw-r--r-- 1 zzg ship 20225774056 Nov 16 23:16 AD01_QC.fastq.1 -rw-r--r-- 1 zzg ship 20225774056 Nov 16 23:17 AD01_QC.fastq.2 -rw-r--r-- 1 zzg ship 26015389018 Nov 16 23:19 AD02_QC.fastq.1 -rw-r--r-- 1 zzg ship 26015389018 Nov 16 23:20 AD02_QC.fastq.2 -rw-r--r-- 1 zzg ship 43305841961 Nov 16 23:23 AD03_QC.fastq.1 -rw-r--r-- 1 zzg ship 43305841961 Nov 16 23:26 AD03_QC.fastq.2 [zzg@manger-server in]$

The size of these three libs are 14.22, 18.29, and 31.23 Gb (90 bp length paired reads), respectively.

We can see metAMOS.logfile. I did not sure whether this issue is related to memory size or not because my memory size is limitted to 32 Gb.

[zzg@manger-server Paired]$ cat metAMOS.logfile Project dir /lustre/user/zzg/Project_ZhangZG/Paired/Cow_metAMOS_test successfully created! Use runPipeline.py to start Pipeline Could not import psutil, disabling. Could not import psutil, disabling. project dir does not exist! usage: runPipeline [options] -d projectdir -h = : print help [this message] -j = : just output all of the programs and citations then exit (default = NO) -v = : verbose output? (default = NO) -d = : directory created by initPipeline (REQUIRED)

[options]: [pipeline_opts] [misc_opts]

[pipeline_opts]: options that affect the pipeline execution Pipeline consists of the following steps: Preprocess, Assemble, FindORFS, MapReads, Abundance, Annotate, FunctionalAnnotation, Scaffold, Propagate, Classify, Postprocess Each of these steps can be referred to by the following options: -f = : force this step to be run (default = NONE) -s = : start at this step in the pipeline (default = Preprocess) -e = : end at this step in the pipeline (default = Postprocess) -n = : step to skip in pipeline (default=NONE)

For each step you can fine-tune the execution as follows [Preprocess] -t = : filter input reads? (default = NO) -q = : produce FastQC quality report for reads with quality information (fastq or sff)? (default = NO) [Assemble] -a = : genome assembler to use (default = SOAPdenovo) -k = : k-mer size to be used for assembly (default = 31) -o = >: min overlap length [MapReads] -m = : read mapper to use? (default = bowtie) -i = : save bowtie (i)ndex? (default = NO) -b = : create library specific per bp coverage of assembled contigs (default = NO) [FindORFS] -g = : gene caller to use (default=FragGeneScan) -l = : min contig length to use for ORF call (default = 300) -x = >: min contig coverage to use for ORF call (default = 3X) [Annotate] -c = : classifier to use for annotation (default = FCP) -u = : annotate unassembled reads? (default = NO) [Classify] -z = : taxonomic level to categorize at (default = class)

[misc_opts]: Miscellaneous options -B = : blast DBs not available (default = NO) -r = : retain the AMOS bank? (default = NO) -p = : number of threads to use (be greedy!) (default=1) -4 = : 454 data? (default = NO) -L = : generate local Krona plots. Local Krona plots can only be viewed on the machine they are generated on but will work on a system with no internet connection (default = NO) [zzg@manger-server Paired]$

  • Zhigang Zhang

Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-65199268 Fax: 86-871-65199318

From: Sergey Koren Date: 2013-11-20 06:21 To: treangen/metAMOS CC: metamoszhang Subject: Re: [metAMOS] Could not import psutil, disabling - runPipeline (#90) Hi,

The frozen binary should include both soapdenovo and metavelvet in the distribution. metAMOS will also continue as long as one of the assemblers was successful. If all assemblies failed, then metAMOS will stop since it cannot continue. The output from runPipeline should indicate the exact error that was encountered which caused the pipeline to stop. Could you send that output along with the assemble step output (found in /Logs/ASSEMBLE.log).

Sergey On Nov 16, 2013, at 11:08 AM, metamoszhang notifications@github.com wrote:

Hi Sergey Koren,

I am trying to test metAMOS (binary release) using my filtered high-quality dataset (167Gb fastq files) including three shortpaired reads libs. Please see below: Lib1: AD01_QC.fastq.1, AD01_QC.fastq.2; Lib2: AD02_QC.fastq.1, AD02_QC.fastq.2; Lib3: AD03_QC.fastq.1, AD03_QC.fastq.2. They were placed within the directory: /lustre/user/zzg/Project_ZhangZG/Paired.

List part of my data:

[zzg@manger-server in]$ head AD01_QC.fastq.1 @HWI-964:267:D23ARACXX:6:1101:1144:2168 1:N:0:GGCTAC AATAATGGTCATCAGTGAAGCCATAAGCCTCACGCTTCATCACCTTAATTTTTGTAAAAATGAATGGAGAAAGCTGACAATCCATTATCT

  • FBH@>EGGFDHHC9GCEF?EC>@CFHG@GGHIIGGGFHGGH @FHGH4=FHFGHIDHGDE>?EHFFCE?B@ACCCCCCCCCCCC:@CCD@C @HWI-964:267:D23ARACXX:6:1101:1207:2228 1:N:0:GGCTAC TGCCTCCGCAGGATGACAAGACGCAGAAGAAAAATCCTGCGGTTAAGGCTCAGCCTACGCTACAAATAGCGGACGACACCATCCCCGATT

  • FHGBHGCFHACF?EEGHII;G<F<HGG>@@FH@HIIGIBG'5:;B7;B>CCECC@CCCC <=CCAC>:ABB0<BB5&5?8@CA<<@.5@ @HWI-964:267:D23ARACXX:6:1101:1431:2192 1:N:0:GGCTAC TGACGATGTGGAGGGCGATGAGGCTTTGGCACATTTCTCGATGAAGGTGCATCAAGACGGATTCCCGAAAGAAGAGTATCCGCAGATTAA [zzg@manger-server in]$ head AD01_QC.fastq.2 @HWI-964:267:D23ARACXX:6:1101:1144:2168 2:N:0:GGCTAC TGGTCACAACCACAAACACGGCATTTGATGCGTTGAACGCGCATGTCGATGTAGACCGCCTTGCGACCGATCGGAACACAACGGAAACGA

  • ?FFAFBHIIIIIIIIIIIIIIIIFIBGHBHGIFGIGIIIIBEEF?CCA=@DBEDDCDDDDDDDDD<BDB@D

    BDDDD?CBBDDBD@BCD@ @HWI-964:267:D23ARACXX:6:1101:1207:2228 2:N:0:GGCTAC GTGCATATCGGCAAACGAGAATTTCTCCTTGCCCTGCGTGGCAACATTCTCTGACTCCTTCTTGCGGAAGAAAGCCTCGCGCTCGCGACG

  • =FHGIIGHIIIBDHIIBDGG0BDGGDFH@FGCG CHDGEHA<CCEEECDDDFFAEEEEEDCDDDDDDB?>BCCC(<CC@DDD<@@DDBB @HWI-964:267:D23ARACXX:6:1101:1431:2192 2:N:0:GGCTAC TTGCCCATCGGCTCGTTCTCCGGCGTAATAGCCCAGAAGCGGATACCGGCAAATTCATATGCCTTCAGGTACTTCACAAAATAGCGTGCA

Now, I used PBS script for running metAMOS in my supercompuper system:

!/bin/sh

PBS -N metaAMOS

PBS -q huge

PBS -l nodes=1:ppn=6

PBS -o /lustre/user/zzg/output/metAMOS-rumen.out

PBS -e /lustre/user/zzg/output/metAMOS-rumen.err

cd $PBS_O_WORKDIR module load metAMOS

cd /lustre/user/zzg/Project_ZhangZG/Paired

initPipeline -1 AD01_QC.fastq.1,AD02_QC.fastq.1,AD03_QC.fastq.1 -2 AD01_QC.fastq.2,AD02_QC.fastq.2,AD03_QC.fastq.2 -d Cow_metAMOS_test -i 300:500,300:500,300:500 runPipeline -a SOAPdenovo,metavelvet -k 31 -p 6 -i YES -c phylosift -d Cow_metAMOS_test | tee metAMOS.logfile

I found initPipeline can sussessfully creat Project dir /lustre/user/zzg/Project_ZhangZG/Paired/Cow_metAMOS_test and can read my data to Preprocess step. You can see the result below:

[zzg@manger-server in]$ pwd /lustre/user/zzg/Project_ZhangZG/Paired/Cow_metAMOS_test/Preprocess/in [zzg@manger-server in]$ ll total 174896600 -rw-r--r-- 1 zzg ship 20225774056 Nov 16 23:16 AD01_QC.fastq.1 -rw-r--r-- 1 zzg ship 20225774056 Nov 16 23:17 AD01_QC.fastq.2 -rw-r--r-- 1 zzg ship 26015389018 Nov 16 23:19 AD02_QC.fastq.1 -rw-r--r-- 1 zzg ship 26015389018 Nov 16 23:20 AD02_QC.fastq.2 -rw-r--r-- 1 zzg ship 43305841961 Nov 16 23:23 AD03_QC.fastq.1 -rw-r--r-- 1 zzg ship 43305841961 Nov 16 23:26 AD03_QC.fastq.2 [zzg@manger-server in]$

However, I found runPipeline would be stopped when running Assemble step becuase I did not find any input and output in Assemble directory. Here, I did not know what will result in its stop. One possibility may be due to lack of genome assembler (whether its installation was performed by user or not ) in binary release. Or please tell me other possible reasons. Many thanks.

Zhigang Zhang

On Thu, Nov 14, 2013 at 10:45 AM, Sergey Koren notifications@github.comwrote:

The binary is self contained so it does not require any python to run. Python and its dependencies is unpacked when the runPipeline binary runs. The only dependencies it has is on perl 5.8.8+ and java 1.6+. So you should be fine to run on your system.

Sergey

On Nov 13, 2013, at 9:06 PM, metamoszhang notifications@github.com wrote:

Hi Sergey ,

Many thanks for your response. I got it. Now, I want to know which python version has been included binary release. My python is 2.7.3. I think, my python should no problem for running metAMOS.

  • Zhigang Zhang

Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-65199268 Fax: 86-871-65199318

From: Sergey Koren Date: 2013-11-14 09:31 To: treangen/metAMOS CC: metamoszhang Subject: Re: [metAMOS] Could not import psutil, disabling - runPipeline (#90) Hi,

Psutil should be included in the binary release but this could be a system-specific issue causing the warning. Installing psutils on your system python will not change the version of python/packages being used by the binary release since it is fully self-contained. However, psutil is only used to estimate available CPU/memory so as long as you specify the number of cpus to use for the pipeline and have at least 32GB of ram available, you should be able to use metAMOS without issues.

Sergey

On Nov 12, 2013, at 10:24 PM, metamoszhang notifications@github.com wrote:

Hi,

I am using a frozen binary MetAMOS with all BLAST DBs. But when I test runPipeline command, an error "Could not import psutil, disabling." was reported. Now, I used Python 2.7.3 and tried to install psutil-0.5.1, psutil 0.6.1, and psutil-1.1.3. But the error was always reported. Please help me resolve it.

My python: [zzg@fat-node-1 ~]$ python Python 2.7.3 (default, Jun 6 2013, 17:48:01) [GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

psutil install:

[zzg@fat-node-1 psutil-0.5.1]$ python setup.py install running install running bdist_egg running egg_info writing psutil.egg-info/PKG-INFO writing top-level names to psutil.egg-info/top_level.txt writing dependency_links to psutil.egg-info/dependency_links.txt reading manifest file 'psutil.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '.html' under directory 'docs' warning: no files found matching '.dia' under directory 'docs' warning: no files found matching '*.png' under directory 'docs' writing manifest file 'psutil.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/psutil copying psutil/psposix.py -> build/lib.linux-x8664-2.7/psutil copying psutil/common.py -> build/lib.linux-x8664-2.7/psutil copying psutil/init.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psmswindows.py -> build/lib.linux-x8664-2.7/psutil copying psutil/pslinux.py -> build/lib.linux-x8664-2.7/psutil copying psutil/error.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psosx.py -> build/lib.linux-x8664-2.7/psutil copying psutil/psbsd.py -> build/lib.linux-x8664-2.7/psutil copying psutil/compat.py -> build/lib.linux-x8664-2.7/psutil running build_ext building 'psutillinux' extension creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/psutil gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutillinux.c -o build/temp.linux-x86_64-2.7/psutil/psutillinux.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutillinux.o -o build/lib.linux-x86_64-2.7/psutillinux.so building 'psutilposix' extension gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutilposix.c -o build/temp.linux-x86_64-2.7/psutil/psutilposix.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutilposix.o -o build/lib.linux-x86_64-2.7/psutilposix.so creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutilposix.so -> build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutillinux.so -> build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psposix.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/common.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/init.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psmswindows.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/pslinux.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/error.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psosx.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psbsd.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/compat.py -> build/bdist.linux-x8664/egg/psutil byte-compiling build/bdist.linux-x86_64/egg/psutil/psposix.py to _psposix.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/common.py to _common.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psmswindows.py to _psmswindows.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/pslinux.py to _pslinux.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/error.py to error.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psosx.py to _psosx.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/psbsd.py to _psbsd.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/compat.py to _compat.pyc creating stub loader for _psutillinux.so creating stub loader for psutilposix.so byte-compiling build/bdist.linux-x86_64/egg/psutillinux.py to psutillinux.pyc byte-compiling build/bdist.linux-x86_64/egg/psutilposix.py to psutilposix.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt zip_safe flag not set; analyzing archive contents... creating dist creating 'dist/psutil-0.5.1-py2.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing psutil-0.5.1-py2.7-linux-x86_64.egg Copying psutil-0.5.1-py2.7-linux-x86_64.egg to /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages Removing psutil 0.6.1 from easy-install.pth file Adding psutil 0.5.1 to easy-install.pth file

Installed /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages/psutil-0.5.1-py2.7-linux-x86_64.egg

Processing dependencies for psutil==0.5.1 Finished processing dependencies for psutil==0.5.1

runPipeline test:

[zzg@fat-node-1 ~]$ runPipeline -h Could not import psutil, disabling. Could not import psutil, disabling. usage: runPipeline [options] -d projectdir -h = : print help [this message] -j = : just output all of the programs and citations then exit (default = NO) -v = : verbose output? (default = NO) -d = : directory created by initPipeline (REQUIRED)

[options]: [pipeline_opts] [misc_opts]

[pipeline_opts]: options that affect the pipeline execution Pipeline consists of the following steps: Preprocess, Assemble, FindORFS, MapReads, Abundance, Annotate, FunctionalAnnotation, Scaffold, Propagate, Classify, Postprocess Each of these steps can be referred to by the following options: -f = : force this step to be run (default = NONE) -s = : start at this step in the pipeline (default = Preprocess) -e = : end at this step in the pipeline (default = Postprocess) -n = : step to skip in pipeline (default=NONE)

For each step you can fine-tune the execution as follows [Preprocess] -t = : filter input reads? (default = NO) -q = : produce FastQC quality report for reads with quality information (fastq or sff)? (default = NO) [Assemble] -a = : genome assembler to use (default = SOAPdenovo) -k = : k-mer size to be used for assembly (default = 31) -o = >: min overlap length [MapReads] -m = : read mapper to use? (default = bowtie) -i = : save bowtie (i)ndex? (default = NO) -b = : create library specific per bp coverage of assembled contigs (default = NO) [FindORFS] -g = : gene caller to use (default=FragGeneScan) -l = : min contig length to use for ORF call (default = 300) -x = >: min contig coverage to use for ORF call (default = 3X) [Annotate] -c = : classifier to use for annotation (default = FCP) -u = : annotate unassembled reads? (default = NO) [Classify] -z = : taxonomic level to categorize at (default = class)

[misc_opts]: Miscellaneous options -B = : blast DBs not available (default = NO) -r = : retain the AMOS bank? (default = NO) -p = : number of threads to use (be greedy!) (default=1) -4 = : 454 data? (default = NO) -L = : generate local Krona plots. Local Krona plots can only be viewed on the machine they are generated on but will work on a system with no internet connection (default = NO) [zzg@fat-node-1 ~]$

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub. — Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/treangen/metAMOS/issues/90#issuecomment-28455215 .

Zhigang Zhang Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-5199268 Fax: 86-871-5199318 — Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub. — Reply to this email directly or view it on GitHub.

metamoszhang commented 10 years ago

Hi Sergey,

Im my system, I has also installed and added SOAPdenovo-63mer and SOAPdenovo-127mer to my $PATH. Those two programs can be run directly without absolute path. Thus, I want to know whether such setup will have negative effects on starting metAMOS assemble step. Otherwise, I did not know what would result in the stop of metAMOS in Assemble step. Please let me share your opinion.

Thank you very much.

Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-65199268 Fax: 86-871-65199318

From: Sergey Koren Date: 2013-11-20 06:21 To: treangen/metAMOS CC: metamoszhang Subject: Re: [metAMOS] Could not import psutil, disabling - runPipeline (#90) Hi,

The frozen binary should include both soapdenovo and metavelvet in the distribution. metAMOS will also continue as long as one of the assemblers was successful. If all assemblies failed, then metAMOS will stop since it cannot continue. The output from runPipeline should indicate the exact error that was encountered which caused the pipeline to stop. Could you send that output along with the assemble step output (found in /Logs/ASSEMBLE.log).

Sergey On Nov 16, 2013, at 11:08 AM, metamoszhang notifications@github.com wrote:

Hi Sergey Koren,

I am trying to test metAMOS (binary release) using my filtered high-quality dataset (167Gb fastq files) including three shortpaired reads libs. Please see below: Lib1: AD01_QC.fastq.1, AD01_QC.fastq.2; Lib2: AD02_QC.fastq.1, AD02_QC.fastq.2; Lib3: AD03_QC.fastq.1, AD03_QC.fastq.2. They were placed within the directory: /lustre/user/zzg/Project_ZhangZG/Paired.

List part of my data:

[zzg@manger-server in]$ head AD01_QC.fastq.1 @HWI-964:267:D23ARACXX:6:1101:1144:2168 1:N:0:GGCTAC AATAATGGTCATCAGTGAAGCCATAAGCCTCACGCTTCATCACCTTAATTTTTGTAAAAATGAATGGAGAAAGCTGACAATCCATTATCT

  • FBH@>EGGFDHHC9GCEF?EC>@CFHG@GGHIIGGGFHGGH @FHGH4=FHFGHIDHGDE>?EHFFCE?B@ACCCCCCCCCCCC:@CCD@C @HWI-964:267:D23ARACXX:6:1101:1207:2228 1:N:0:GGCTAC TGCCTCCGCAGGATGACAAGACGCAGAAGAAAAATCCTGCGGTTAAGGCTCAGCCTACGCTACAAATAGCGGACGACACCATCCCCGATT

  • FHGBHGCFHACF?EEGHII;G<F<HGG>@@FH@HIIGIBG'5:;B7;B>CCECC@CCCC <=CCAC>:ABB0<BB5&5?8@CA<<@.5@ @HWI-964:267:D23ARACXX:6:1101:1431:2192 1:N:0:GGCTAC TGACGATGTGGAGGGCGATGAGGCTTTGGCACATTTCTCGATGAAGGTGCATCAAGACGGATTCCCGAAAGAAGAGTATCCGCAGATTAA [zzg@manger-server in]$ head AD01_QC.fastq.2 @HWI-964:267:D23ARACXX:6:1101:1144:2168 2:N:0:GGCTAC TGGTCACAACCACAAACACGGCATTTGATGCGTTGAACGCGCATGTCGATGTAGACCGCCTTGCGACCGATCGGAACACAACGGAAACGA

  • ?FFAFBHIIIIIIIIIIIIIIIIFIBGHBHGIFGIGIIIIBEEF?CCA=@DBEDDCDDDDDDDDD<BDB@D

    BDDDD?CBBDDBD@BCD@ @HWI-964:267:D23ARACXX:6:1101:1207:2228 2:N:0:GGCTAC GTGCATATCGGCAAACGAGAATTTCTCCTTGCCCTGCGTGGCAACATTCTCTGACTCCTTCTTGCGGAAGAAAGCCTCGCGCTCGCGACG

  • =FHGIIGHIIIBDHIIBDGG0BDGGDFH@FGCG CHDGEHA<CCEEECDDDFFAEEEEEDCDDDDDDB?>BCCC(<CC@DDD<@@DDBB @HWI-964:267:D23ARACXX:6:1101:1431:2192 2:N:0:GGCTAC TTGCCCATCGGCTCGTTCTCCGGCGTAATAGCCCAGAAGCGGATACCGGCAAATTCATATGCCTTCAGGTACTTCACAAAATAGCGTGCA

Now, I used PBS script for running metAMOS in my supercompuper system:

!/bin/sh

PBS -N metaAMOS

PBS -q huge

PBS -l nodes=1:ppn=6

PBS -o /lustre/user/zzg/output/metAMOS-rumen.out

PBS -e /lustre/user/zzg/output/metAMOS-rumen.err

cd $PBS_O_WORKDIR module load metAMOS

cd /lustre/user/zzg/Project_ZhangZG/Paired

initPipeline -1 AD01_QC.fastq.1,AD02_QC.fastq.1,AD03_QC.fastq.1 -2 AD01_QC.fastq.2,AD02_QC.fastq.2,AD03_QC.fastq.2 -d Cow_metAMOS_test -i 300:500,300:500,300:500 runPipeline -a SOAPdenovo,metavelvet -k 31 -p 6 -i YES -c phylosift -d Cow_metAMOS_test | tee metAMOS.logfile

I found initPipeline can sussessfully creat Project dir /lustre/user/zzg/Project_ZhangZG/Paired/Cow_metAMOS_test and can read my data to Preprocess step. You can see the result below:

[zzg@manger-server in]$ pwd /lustre/user/zzg/Project_ZhangZG/Paired/Cow_metAMOS_test/Preprocess/in [zzg@manger-server in]$ ll total 174896600 -rw-r--r-- 1 zzg ship 20225774056 Nov 16 23:16 AD01_QC.fastq.1 -rw-r--r-- 1 zzg ship 20225774056 Nov 16 23:17 AD01_QC.fastq.2 -rw-r--r-- 1 zzg ship 26015389018 Nov 16 23:19 AD02_QC.fastq.1 -rw-r--r-- 1 zzg ship 26015389018 Nov 16 23:20 AD02_QC.fastq.2 -rw-r--r-- 1 zzg ship 43305841961 Nov 16 23:23 AD03_QC.fastq.1 -rw-r--r-- 1 zzg ship 43305841961 Nov 16 23:26 AD03_QC.fastq.2 [zzg@manger-server in]$

However, I found runPipeline would be stopped when running Assemble step becuase I did not find any input and output in Assemble directory. Here, I did not know what will result in its stop. One possibility may be due to lack of genome assembler (whether its installation was performed by user or not ) in binary release. Or please tell me other possible reasons. Many thanks.

Zhigang Zhang

On Thu, Nov 14, 2013 at 10:45 AM, Sergey Koren notifications@github.comwrote:

The binary is self contained so it does not require any python to run. Python and its dependencies is unpacked when the runPipeline binary runs. The only dependencies it has is on perl 5.8.8+ and java 1.6+. So you should be fine to run on your system.

Sergey

On Nov 13, 2013, at 9:06 PM, metamoszhang notifications@github.com wrote:

Hi Sergey ,

Many thanks for your response. I got it. Now, I want to know which python version has been included binary release. My python is 2.7.3. I think, my python should no problem for running metAMOS.

  • Zhigang Zhang

Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-65199268 Fax: 86-871-65199318

From: Sergey Koren Date: 2013-11-14 09:31 To: treangen/metAMOS CC: metamoszhang Subject: Re: [metAMOS] Could not import psutil, disabling - runPipeline (#90) Hi,

Psutil should be included in the binary release but this could be a system-specific issue causing the warning. Installing psutils on your system python will not change the version of python/packages being used by the binary release since it is fully self-contained. However, psutil is only used to estimate available CPU/memory so as long as you specify the number of cpus to use for the pipeline and have at least 32GB of ram available, you should be able to use metAMOS without issues.

Sergey

On Nov 12, 2013, at 10:24 PM, metamoszhang notifications@github.com wrote:

Hi,

I am using a frozen binary MetAMOS with all BLAST DBs. But when I test runPipeline command, an error "Could not import psutil, disabling." was reported. Now, I used Python 2.7.3 and tried to install psutil-0.5.1, psutil 0.6.1, and psutil-1.1.3. But the error was always reported. Please help me resolve it.

My python: [zzg@fat-node-1 ~]$ python Python 2.7.3 (default, Jun 6 2013, 17:48:01) [GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2 Type "help", "copyright", "credits" or "license" for more information.

psutil install:

[zzg@fat-node-1 psutil-0.5.1]$ python setup.py install running install running bdist_egg running egg_info writing psutil.egg-info/PKG-INFO writing top-level names to psutil.egg-info/top_level.txt writing dependency_links to psutil.egg-info/dependency_links.txt reading manifest file 'psutil.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '.html' under directory 'docs' warning: no files found matching '.dia' under directory 'docs' warning: no files found matching '*.png' under directory 'docs' writing manifest file 'psutil.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/psutil copying psutil/psposix.py -> build/lib.linux-x8664-2.7/psutil copying psutil/common.py -> build/lib.linux-x8664-2.7/psutil copying psutil/init.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psmswindows.py -> build/lib.linux-x8664-2.7/psutil copying psutil/pslinux.py -> build/lib.linux-x8664-2.7/psutil copying psutil/error.py -> build/lib.linux-x86_64-2.7/psutil copying psutil/psosx.py -> build/lib.linux-x8664-2.7/psutil copying psutil/psbsd.py -> build/lib.linux-x8664-2.7/psutil copying psutil/compat.py -> build/lib.linux-x8664-2.7/psutil running build_ext building 'psutillinux' extension creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/psutil gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutillinux.c -o build/temp.linux-x86_64-2.7/psutil/psutillinux.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutillinux.o -o build/lib.linux-x86_64-2.7/psutillinux.so building 'psutilposix' extension gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/lustre/user/zzg/pub/Python-2.7.3/include/python2.7 -c psutil/psutilposix.c -o build/temp.linux-x86_64-2.7/psutil/psutilposix.o gcc -pthread -shared build/temp.linux-x86_64-2.7/psutil/psutilposix.o -o build/lib.linux-x86_64-2.7/psutilposix.so creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutilposix.so -> build/bdist.linux-x86_64/egg copying build/lib.linux-x86_64-2.7/psutillinux.so -> build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psposix.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/common.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/init.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psmswindows.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/pslinux.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/error.py -> build/bdist.linux-x86_64/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psosx.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/psbsd.py -> build/bdist.linux-x8664/egg/psutil copying build/lib.linux-x86_64-2.7/psutil/compat.py -> build/bdist.linux-x8664/egg/psutil byte-compiling build/bdist.linux-x86_64/egg/psutil/psposix.py to _psposix.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/common.py to _common.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/init.py to init.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psmswindows.py to _psmswindows.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/pslinux.py to _pslinux.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/error.py to error.pyc byte-compiling build/bdist.linux-x86_64/egg/psutil/psosx.py to _psosx.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/psbsd.py to _psbsd.pyc byte-compiling build/bdist.linux-x8664/egg/psutil/compat.py to _compat.pyc creating stub loader for _psutillinux.so creating stub loader for psutilposix.so byte-compiling build/bdist.linux-x86_64/egg/psutillinux.py to psutillinux.pyc byte-compiling build/bdist.linux-x86_64/egg/psutilposix.py to psutilposix.pyc creating build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying psutil.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt zip_safe flag not set; analyzing archive contents... creating dist creating 'dist/psutil-0.5.1-py2.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing psutil-0.5.1-py2.7-linux-x86_64.egg Copying psutil-0.5.1-py2.7-linux-x86_64.egg to /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages Removing psutil 0.6.1 from easy-install.pth file Adding psutil 0.5.1 to easy-install.pth file

Installed /lustre/user/zzg/pub/Python-2.7.3/lib/python2.7/site-packages/psutil-0.5.1-py2.7-linux-x86_64.egg

Processing dependencies for psutil==0.5.1 Finished processing dependencies for psutil==0.5.1

runPipeline test:

[zzg@fat-node-1 ~]$ runPipeline -h Could not import psutil, disabling. Could not import psutil, disabling. usage: runPipeline [options] -d projectdir -h = : print help [this message] -j = : just output all of the programs and citations then exit (default = NO) -v = : verbose output? (default = NO) -d = : directory created by initPipeline (REQUIRED)

[options]: [pipeline_opts] [misc_opts]

[pipeline_opts]: options that affect the pipeline execution Pipeline consists of the following steps: Preprocess, Assemble, FindORFS, MapReads, Abundance, Annotate, FunctionalAnnotation, Scaffold, Propagate, Classify, Postprocess Each of these steps can be referred to by the following options: -f = : force this step to be run (default = NONE) -s = : start at this step in the pipeline (default = Preprocess) -e = : end at this step in the pipeline (default = Postprocess) -n = : step to skip in pipeline (default=NONE)

For each step you can fine-tune the execution as follows [Preprocess] -t = : filter input reads? (default = NO) -q = : produce FastQC quality report for reads with quality information (fastq or sff)? (default = NO) [Assemble] -a = : genome assembler to use (default = SOAPdenovo) -k = : k-mer size to be used for assembly (default = 31) -o = >: min overlap length [MapReads] -m = : read mapper to use? (default = bowtie) -i = : save bowtie (i)ndex? (default = NO) -b = : create library specific per bp coverage of assembled contigs (default = NO) [FindORFS] -g = : gene caller to use (default=FragGeneScan) -l = : min contig length to use for ORF call (default = 300) -x = >: min contig coverage to use for ORF call (default = 3X) [Annotate] -c = : classifier to use for annotation (default = FCP) -u = : annotate unassembled reads? (default = NO) [Classify] -z = : taxonomic level to categorize at (default = class)

[misc_opts]: Miscellaneous options -B = : blast DBs not available (default = NO) -r = : retain the AMOS bank? (default = NO) -p = : number of threads to use (be greedy!) (default=1) -4 = : 454 data? (default = NO) -L = : generate local Krona plots. Local Krona plots can only be viewed on the machine they are generated on but will work on a system with no internet connection (default = NO) [zzg@fat-node-1 ~]$

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub. — Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/treangen/metAMOS/issues/90#issuecomment-28455215 .

Zhigang Zhang Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-5199268 Fax: 86-871-5199318 — Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub.

skoren commented 10 years ago

The version of SOAPdenovo you have installed in your path does not matter. metAMOS will prefer to use its own version of a tool if it can be found. Only if the included version of a tool is not compatible with your system will it try to look for it in your path.

Your metamos run never started, it did not stop in the Assemble step. The Preprocess/in directory is created by initPipeline. The parameters specified to runPipeline were bad, -i does not expect any arguments. As a result, runPipeline reported an error and never started. You can see this in your output: project dir does not exist! usage: runPipeline [options] -d projectdir Your runPipeline command should be updated to: runPipeline -k 51 -p 6 -i -c phylosift -d Cow_metAMOS_test | tee metAMOS.logfile not what you are currently using: runPipeline -k 51 -p 6 -i YES -c phylosift -d Cow_metAMOS_test | tee metAMOS.logfile

Also, as I mentioned previously, the dataset you are using is likely the largest dataset metAMOS has been run on. The assemble step will take a large amount of RAM, especially for metavelvet. I would definitely say 32GB Is not going to be enough. The preprocess and mapread steps will likely be slow. I would also recommend not using phylosift as a classifier as it can be slow on large datasets, instead I would recommend kraken.

Sergey

metamoszhang commented 10 years ago

Dear Sergey,

Many thanks for your replies. I got it and will try your thoughts using my data. Indeed, my data is so large that 512G RAM is not enough to run metvelvet. Only I can choose SOAPdenovo with lower RAM requirement.

Thanks again.

Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-65199268 Fax: 86-871-65199318

From: Sergey Koren Date: 2013-11-22 06:06 To: treangen/metAMOS CC: metamoszhang Subject: Re: [metAMOS] Could not import psutil, disabling - runPipeline (#90) The version of SOAPdenovo you have installed in your path does not matter. metAMOS will prefer to use its own version of a tool if it can be found. Only if the included version of a tool is not compatible with your system will it try to look for it in your path. Your metamos run never started, it did not stop in the Assemble step. The Preprocess/in directory is created by initPipeline. The parameters specified to runPipeline were bad, -i does not expect any arguments. As a result, runPipeline reported an error and never started. You can see this in your output: project dir does not exist! usage: runPipeline [options] -d projectdir Your runPipeline command should be updated to: runPipeline -k 51 -p 6 -i -c phylosift -d Cow_metAMOS_test | tee metAMOS.logfile not what you are currently using: runPipeline -k 51 -p 6 -i YES -c phylosift -d Cow_metAMOS_test | tee metAMOS.logfile Also, as I mentioned previously, the dataset you are using is likely the largest dataset metAMOS has been run on. The assemble step will take a large amount of RAM, especially for metavelvet. I would definitely say 32GB Is not going to be enough. The preprocess and mapread steps will likely be slow. I would also recommend not using phylosift as a classifier as it can be slow on large datasets, instead I would recommend kraken. Sergey — Reply to this email directly or view it on GitHub.

metamoszhang commented 10 years ago

Dear Sergey,

According to your suggestion, I run the below script. But Assemble process is not still started. Please see the attached result.summary file and help me find which reason would stop Assembling.

!/bin/sh

PBS -N metAMOS

PBS -q huge

PBS -l nodes=1:ppn=4

PBS -o /lustre/user/zzg/output/metAMOS-rumen.out

PBS -e /lustre/user/zzg/output/metAMOS-rumen.err

cd $PBS_O_WORKDIR module load metAMOS

cd /lustre/user/zzg/Project_ZhangZG/Paired

initPipeline -1 AD01_QC.fastq.1 -2 AD01_QC.fastq.2 -d Yak_metAMOS_test -i 300:500 runPipeline -k 41 -p 6 -i -c kraken -d Yak_metAMOS_test | tee metAMOS.logfile ### -a SOAPdenovo,metavelvet

In my opinion, the results outputted from Preprocess step should be right to run SOAPdenovo Assembling, such as using lib1.fasta file or lib1.1.fasta and lib1.2.fasta together.

[zzg@fat-node-1 Preprocess]$ cd out/ [zzg@fat-node-1 out]$ ll total 304435356 -rw-r--r-- 3 zzg ship 20225774056 Nov 25 10:27 AD01_QC.fastq.1 -rw-r--r-- 3 zzg ship 20225774056 Nov 25 10:28 AD01_QC.fastq.2 -rw-r--r-- 1 zzg ship 12423412296 Nov 25 20:29 lib1.1.fasta -rw-r--r-- 1 zzg ship 23301228137 Nov 25 20:29 lib1.1.fasta.qual -rw-r--r-- 3 zzg ship 20225774056 Nov 25 10:27 lib1.1.fastq -rw-r--r-- 1 zzg ship 12423412296 Nov 25 23:30 lib1.2.fasta -rw-r--r-- 1 zzg ship 23301087923 Nov 25 23:30 lib1.2.fasta.qual -rw-r--r-- 3 zzg ship 20225774056 Nov 25 10:28 lib1.2.fastq -rw-r--r-- 1 zzg ship 24846824592 Nov 25 17:25 lib1.fasta -rw-r--r-- 1 zzg ship 46602316060 Nov 25 17:25 lib1.fasta.qual -rw-r--r-- 2 zzg ship 40451548112 Nov 25 10:37 lib1.fastq -rw-r--r-- 2 zzg ship 40451548112 Nov 25 10:37 lib1.seq -rw-r--r-- 1 zzg ship 7037085792 Nov 25 11:11 lib1.seq.mates -rw-r--r-- 1 zzg ship 0 Nov 25 23:30 preprocess.success

If running SOAPdenovo based config.txt, I think, two files lib1.1.fastq and lib1.2.fastq should also be appropriate.

[zzg@manger-server Yak_metAMOS_test]$ more config.txt max_rd_len=150 [LIB] avg_ins=400.0 reverse_seq=0 asm_flags=3 rank=1 q1=LIB1Q1REPLACE q2=LIB1Q2REPLACE

Here, it is very difficult to understand why SOAPdenovo assembling was not started. Please give me some clues for resolving this issue.

Many thanks

Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-65199268 Fax: 86-871-65199318

From: Sergey Koren Date: 2013-11-22 06:06 To: treangen/metAMOS CC: metamoszhang Subject: Re: [metAMOS] Could not import psutil, disabling - runPipeline (#90) The version of SOAPdenovo you have installed in your path does not matter. metAMOS will prefer to use its own version of a tool if it can be found. Only if the included version of a tool is not compatible with your system will it try to look for it in your path. Your metamos run never started, it did not stop in the Assemble step. The Preprocess/in directory is created by initPipeline. The parameters specified to runPipeline were bad, -i does not expect any arguments. As a result, runPipeline reported an error and never started. You can see this in your output: project dir does not exist! usage: runPipeline [options] -d projectdir Your runPipeline command should be updated to: runPipeline -k 51 -p 6 -i -c phylosift -d Cow_metAMOS_test | tee metAMOS.logfile not what you are currently using: runPipeline -k 51 -p 6 -i YES -c phylosift -d Cow_metAMOS_test | tee metAMOS.logfile Also, as I mentioned previously, the dataset you are using is likely the largest dataset metAMOS has been run on. The assemble step will take a large amount of RAM, especially for metavelvet. I would definitely say 32GB Is not going to be enough. The preprocess and mapread steps will likely be slow. I would also recommend not using phylosift as a classifier as it can be slow on large datasets, instead I would recommend kraken. Sergey — Reply to this email directly or view it on GitHub.

skoren commented 10 years ago

OK, from the output of Preprocess/out, it does look like runPipeline started. Can you provide the output from runPipeline as well as Logs/ASSEMBLE.log (if it exists).

metamoszhang commented 10 years ago

Hi,

PBS script:

!/bin/sh

PBS -N metAMOS

PBS -q huge

PBS -l nodes=1:fat-node-1:ppn=4

PBS -o /lustre/user/zzg/output/metAMOS-rumen.out

PBS -e /lustre/user/zzg/output/metAMOS-rumen.err

cd $PBS_O_WORKDIR module load metAMOS

cd /lustre/user/zzg/Project_ZhangZG/Paired

initPipeline -1 AD01_QC.fastq.1 -2 AD01_QC.fastq.2 -d Yak_metAMOS_test -i 300:500 runPipeline -k 41 -p 6 -i -c kraken -d Yak_metAMOS_test | tee metAMOS.logfile

After it is over, you can see three files (metAMOS.o36032, metAMOS.e36032, and metAMOS.logfile) from the attached, which recorded all running info.

All output files by runPipeline were listet below:

[zzg@manger-server Yak_metAMOS_test]$ ll total 80 -rw-r--r-- 1 zzg ship 1962 Nov 25 10:29 20131125102900.txt drwxr-xr-x 4 zzg ship 4096 Nov 25 10:26 Abundance drwxr-xr-x 4 zzg ship 4096 Nov 25 10:26 Annotate drwxr-xr-x 4 zzg ship 4096 Nov 25 10:26 Assemble drwxr-xr-x 4 zzg ship 4096 Nov 25 10:26 Classify -rw-r--r-- 1 zzg ship 102 Nov 25 10:28 config.txt drwxr-xr-x 4 zzg ship 4096 Nov 25 10:26 FindORFS drwxr-xr-x 4 zzg ship 4096 Nov 25 10:26 FindRepeats drwxr-xr-x 4 zzg ship 4096 Nov 25 10:26 FindScaffoldORFS drwxr-xr-x 4 zzg ship 4096 Nov 25 10:26 FunctionalAnnotation drwxr-xr-x 2 zzg ship 4096 Nov 25 23:30 Logs drwxr-xr-x 4 zzg ship 4096 Nov 25 10:26 MultiAlign -rw-r--r-- 1 zzg ship 2243 Nov 25 10:29 pipeline.conf -rw-r--r-- 1 zzg ship 203 Nov 25 10:28 pipeline.ini -rw-r--r-- 1 zzg ship 1962 Nov 25 10:29 pipeline.run drwxr-xr-x 4 zzg ship 4096 Nov 25 10:26 Postprocess drwxr-xr-x 4 zzg ship 4096 Nov 25 10:26 Preprocess drwxr-xr-x 4 zzg ship 4096 Nov 25 10:26 Propagate drwxr-xr-x 4 zzg ship 4096 Nov 25 10:26 Scaffold drwxr-xr-x 4 zzg ship 4096 Nov 25 10:26 Validate

Obly Preprocess outputted results files. Please see the below:

[zzg@manger-server Yak_metAMOS_test]$ cd Preprocess/ [zzg@manger-server Preprocess]$ ll total 8 drwxr-xr-x 2 zzg ship 4096 Nov 25 10:27 in drwxr-xr-x 2 zzg ship 4096 Nov 25 23:30 out [zzg@manger-server Preprocess]$ cd in/ [zzg@manger-server in]$ ll total 39503492 -rw-r--r-- 3 zzg ship 20225774056 Nov 25 10:27 AD01_QC.fastq.1 -rw-r--r-- 3 zzg ship 20225774056 Nov 25 10:28 AD01_QC.fastq.2 [zzg@manger-server in]$ cd ../out/ [zzg@manger-server out]$ ll total 304435356 -rw-r--r-- 3 zzg ship 20225774056 Nov 25 10:27 AD01_QC.fastq.1 -rw-r--r-- 3 zzg ship 20225774056 Nov 25 10:28 AD01_QC.fastq.2 -rw-r--r-- 1 zzg ship 12423412296 Nov 25 20:29 lib1.1.fasta -rw-r--r-- 1 zzg ship 23301228137 Nov 25 20:29 lib1.1.fasta.qual -rw-r--r-- 3 zzg ship 20225774056 Nov 25 10:27 lib1.1.fastq -rw-r--r-- 1 zzg ship 12423412296 Nov 25 23:30 lib1.2.fasta -rw-r--r-- 1 zzg ship 23301087923 Nov 25 23:30 lib1.2.fasta.qual -rw-r--r-- 3 zzg ship 20225774056 Nov 25 10:28 lib1.2.fastq -rw-r--r-- 1 zzg ship 24846824592 Nov 25 17:25 lib1.fasta -rw-r--r-- 1 zzg ship 46602316060 Nov 25 17:25 lib1.fasta.qual -rw-r--r-- 2 zzg ship 40451548112 Nov 25 10:37 lib1.fastq -rw-r--r-- 2 zzg ship 40451548112 Nov 25 10:37 lib1.seq -rw-r--r-- 1 zzg ship 7037085792 Nov 25 11:11 lib1.seq.mates -rw-r--r-- 1 zzg ship 0 Nov 25 23:30 preprocess.success

In Assemble directory, yo can see two empty files assemble.success and mapreads.success.

[zzg@manger-server out]$ pwd /lustre/user/zzg/Project_ZhangZG/Paired/Yak_metAMOS_test/Assemble/out [zzg@manger-server out]$ ll total 0 -rw-r--r-- 1 zzg ship 0 Nov 25 23:30 assemble.success -rw-r--r-- 1 zzg ship 0 Nov 25 23:30 mapreads.success [zzg@manger-server out]$

In Logs directory,

ASSEMBLE.log was produced but empty. Other log files produced can be found from the attached files.

[zzg@manger-server Logs]$ ll total 16 -rw-r--r-- 1 zzg ship 0 Nov 25 23:30 ASSEMBLE.log -rw-r--r-- 1 zzg ship 0 Nov 25 23:30 assemble.ok -rw-r--r-- 1 zzg ship 0 Nov 25 23:30 assemble.started -rw-r--r-- 1 zzg ship 5055 Nov 25 23:30 COMMANDS.log -rw-r--r-- 1 zzg ship 0 Nov 25 23:30 mapreads.ok -rw-r--r-- 1 zzg ship 798 Nov 25 23:30 PREPROCESS.log -rw-r--r-- 1 zzg ship 0 Nov 25 23:30 preprocess.ok -rw-r--r-- 1 zzg ship 0 Nov 25 10:29 preprocess.started -rw-r--r-- 1 zzg ship 1292 Nov 25 23:30 VALIDATE.log -rw-r--r-- 1 zzg ship 0 Nov 25 23:30 validate.ok -rw-r--r-- 1 zzg ship 0 Nov 25 23:30 validate.started

Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-65199268 Fax: 86-871-65199318

From: Sergey Koren Date: 2013-11-27 00:39 To: treangen/metAMOS CC: metamoszhang Subject: Re: [metAMOS] Could not import psutil, disabling - runPipeline (#90) OK, from the output of Preprocess/out, it does look like runPipeline started. Can you provide the output from runPipeline as well as Logs/ASSEMBLE.log (if it exists). — Reply to this email directly or view it on GitHub.

skoren commented 10 years ago

The issue is the known bug in release 1.3 where not specifying an assembler causes an error (#89). This is fixed in the latest version of the binary. If you specify an assembler (-a soap,metavelvet) the pipeline should run.

metamoszhang commented 10 years ago

Thank your very much. I will try to test it.

Laboratory of Evolutionary & Functional Genomics Kunming Institute of Zoology Chinese Academy of Sciences 32 Jiaochangdong Road Kunming, Yunnan 650223 P.R.China Phone (office): 86-871-65199268 Fax: 86-871-65199318

From: Sergey Koren Date: 2013-11-28 08:31 To: treangen/metAMOS CC: metamoszhang Subject: Re: [metAMOS] Could not import psutil, disabling - runPipeline (#90) The issue is the known bug in release 1.3 where not specifying an assembler causes an error (#89). This is fixed in the latest version of the binary. If you specify an assembler (-a soap,metavelvet) the pipeline should run. — Reply to this email directly or view it on GitHub.

skoren commented 10 years ago

Closing due to inactivity.