search

hallamlab / metapathways2

MetaPathways v2.0: A master-worker model for environmental Pathway/Genome Database construction on grids and clouds
http://hallam.microbiology.ubc.ca/MetaPathways/
33 stars 14 forks source link

PathoLogic error opaque to user #56

Closed taltman closed 7 years ago

taltman commented 9 years ago

Running MP from the command line, I got the following error when trying to run the PathoLogic step. The problem is that the error message is generic, and doesn't specify what went wrong, and thus there's no way for the user to know what to try.

MP should:

  1. Make it crystal clear that the code is failing in the MP python code, or in Pathway Tools
  2. Bubble up at least part of the error message from PTools to the user via the GUI or CLI
  3. Provide a filesystem path to the corresponding PathoLogic log file for full debugging information.

time /afs/ir/users/t/a/taltman1/farmshare/third-party/bin/MetaPathways/v2.5.2/MetaPathways_Python.2.5.1//libs/python_scripts/MetaPathways_run_pathologic.py ERROR Failed to run Pathologic on input /srv/zfs01/user_data/taltman1/single-cell-assembly-files/metapathways_annots/gtl1/genome/ptools// : INFO Kill any other PathwayTools instance running on the machine and try again ERROR ERROR Failed to run Pathologic on input /srv/zfs01/user_data/taltman1/single-cell-assembly-files/metapathways_annots/gtl1/genome/ptools// : ERROR Exiting the Python code

GiangLeN commented 7 years ago

I also get the same problem as above PROCESSING INPUT ../project/input//spd3_9ctgs.fa [STEPS BLOCK 2] PATHOLOGIC INPUT ..... Failed! RUNNING PATHOLOGIC ERROR ERROR Cannot find all inputs for Pathologic in folder /data/pathway/project/output/spd3_9ctgs/ptools// : ERROR Exiting the Python code

Is there anyway to fix this? Many thanks

taltman commented 7 years ago

Hi Giang,

Can you provide a list of the contents of your /ptools directory? A file manifest?

Thanks,

~T

GiangLeN commented 7 years ago

Hi T, Is this what you are looking for? ptool_content.txt Thanks, G

taltman commented 7 years ago

Nope. I'm asking you to tell us what is in your ptools directory, for your project. Check the wiki for documentation on the project output directory format for more information.

GiangLeN commented 7 years ago

If you mean the folder ptools from the output directory then there is no file in it.

taltman commented 7 years ago

Have you installed Pathway Tools? Have you verified that you can start Pathway Tools without error?

GiangLeN commented 7 years ago

I have installed Pathway Tools but have yet to verified it. I will look into it now. Thanks

taltman commented 7 years ago

Hi Giang, It looks like MetaPathways is failing to generate the output needed as input to Pathway Tools. Why it is failing is not clear to me. One way to get more useful debugging information is to copy & paste the verbatim command that MetaPathways displays to the command line, and run it directly.

ariahahn commented 7 years ago

Hi Giang,

A few more questions. Did you get your databases to work? Do you have a table with results in it in the results folder called functional and taxonomic table? What is the error in the error window and finally, what is the name of your sample?

Thanks,

Aria

On Oct 20, 2016 11:13 PM, "Tomer Altman" notifications@github.com wrote:

Hi Giang, It looks like MetaPathways is failing to generate the output needed as input to Pathway Tools. Why it is failing is not clear to me. One way to get more useful debugging information is to copy & paste the verbatim command that MetaPathways displays to the command line, and run it directly.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hallamlab/metapathways2/issues/56#issuecomment-255302360, or mute the thread https://github.com/notifications/unsubscribe-auth/AI6RqEnzjW-Jkm8PM8yonJsTWPrbWGJmks5q2FgggaJpZM4Dkolj .

GiangLeN commented 7 years ago

Hi Aria,

It takes me awhile to figure out that there was no genbank input and so no pathologic input. For some strange reason my machine could not run MetaPathways_create_genbank.py. I did test this out by changing the template_config.txt as follow: GENBANK_FILE 'libs/python_scripts/MetaPathways_create_genbank_ptinput_sequin.py' With the above line I was be able to run and complete the Pathologic input process.

Right now the program is giving me this error:

eeror lost ("Kill any other PathwayTools instance running on the machine and try again : /data/pathway/pathway-tools/pathway-tools -patho /data/pathway/project/out2.5.1in/spd3_9ctgs/ptools// -no-taxonomic-pruning -no-web-cel-overview -api")

Regarding your questions: I did get some database to work after changing the config file. The one which is not showing is the Metacyc result and greengenes. There is a file called functional and taxonomic table in the report folder. The name of my sample is spd3_9ctgs.

Thank you so much for your help.

Giang

ariahahn commented 7 years ago

Hi Giang,

MetaCyc will only show if you build a pgdb. Greengenes will show if you used greengenes (did you?). So you do have ptools input now? If yes, look into where you have stored any past PGDBs (itll be in your set up window of Metapathways and should be something like /Users/ariahahn/ptools-local/pgdbs/user/samplename.cyc) and delete the samplename.cyc file. Pathwaytools wont rerun if you have anything with the name in that folder. Youll have to do this every time between attempts. Now make sure you dont have pathway tools running (try the "top" command in the command line to be sure - if you're not comfortable with this then simply restart the computer). Another idea would be to make a small test sample with a new name and try that. Let me know how it goes.

Aria

On Wed, Oct 26, 2016 at 12:23 AM, GiangLeN notifications@github.com wrote:

Hi Aria,

It takes me awhile to figure out that there was no genbank input and so no pathologic input. For some strange reason my machine could not run MetaPathways_create_genbank.py. I did test this out by edited the template_config.txt as follow: GENBANK_FILE 'libs/python_scripts/MetaPathways_creategenbank ptinput_sequin.py' With the above line I was be able to run and complete the Pathologic input process.

Right now the program is giving me this error:

eeror lost ("Kill any other PathwayTools instance running on the machine and try again : /data/pathway/pathway-tools/pathway-tools -patho /data/pathway/project/out2.5.1in/spd3_9ctgs/ptools// -no-taxonomic-pruning -no-web-cel-overview -api")

Regarding your questions: I did get some database to work after changing the config file. The one which is not showing is the Metacyc result and greengenes. There is a file called functional and taxonomic table in the report folder. The name of my sample is spd3_9ctgs.

Thank you so much for your help.

Giang

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hallamlab/metapathways2/issues/56#issuecomment-256269283, or mute the thread https://github.com/notifications/unsubscribe-auth/AI6RqApncS-kZY6BPEiqt5aLzDmIzD9pks5q3v_fgaJpZM4Dkolj .

Aria Hahn MSc PhD Candidate www.cupcakesandscience.com Hallam Laboratory http://www.cmde.science.ubc.ca/hallam/index.php

University of British Columbia Department of Microbiology & Immunology Life Sciences Centre 2350 Health Sciences Mall (Rm 2520) Vancouver, BC Canada V6T 1Z3 T: 604-827-4216 F: 604-822-6041 ariahahn@interchange.ubc.ca

GiangLeN commented 7 years ago

Hi Aria,

Thank you for your reply. I do have ptools input now. So does that mean the file Metapathways_create_genbank.py is changeable with the Metapathways_create_genbank_ptinput_sequin.py file? In my directory (/data/pathway/project/ptools-local/pgdbs/user) there are 3 files: PGDB-counter.dat , PGDB-METADATA.ocelot and PGDB-METADATA.ocelot. I have moved them to a new directory for the test run.

Regarding rRNA databases, I downloaded GreenGenes and Silva fasta files and set these databases up together. However, metapathways only displays Silva result. For the test run here is the result (I deleted the RUNNING task to reduce the report length):

============== BEGIN RUN test0 19:24:53 BLOCK1 ================ FUNC_SEARCH:metacyc-v5-2011-10-21 SUCCESS FUNC_SEARCH:COG_2013-12-27 SUCCESS FUNC_SEARCH:CAZY_2014_09_04 SUCCESS ============== BEGIN RUN test0 19:24:53 BLOCK2 ================ COMPUTE_REFSCORES SUCCESS PARSE_FUNC_SEARCH:metacyc-v5-2011-10-21 SUCCESS PARSE_FUNC_SEARCH:COG_2013-12-27 SUCCESS PARSE_FUNC_SEARCH:CAZY_2014_09_04 SUCCESS SCAN_rRNA:SILVA_128_SSURef_Nr99_tax_silva SUCCESS SCAN_rRNA:Greengenes_13_15 SUCCESS SCAN_rRNA:SILVA_128_LSURef_tax_silva SUCCESS SCAN_tRNA SUCCESS ANNOTATE_ORFS SUCCESS GENBANK_FILE SUCCESS PATHOLOGIC_INPUT SUCCESS CREATE_ANNOT_REPORTS SUCCESS BUILD_PGDB RUNNING

The program would fail to build PGDB. I manually activated pathway-tools at the result folder (/data/pathway/project/output2/test0/ptools) as follow: /data/pathway/pathway-tools/pathway-tools -patho ./ -no-taxonomic-pruning -no-web-cel-overview -api

Note: for some strange reason I no longer could recreate the result below:

After that the file pathologic.log file is created and I was able to get a green tick at build PGDB. Here is the log RUNNING PATHOLOGIC
RUNNING MetaPathways version 2.5.2 Fixing file /data/pathway/project/ptools-local/pgdbs/user/test0cyc/1.0/input/organism.dat Starting up pathway tools

Traceback (most recent call last): File "/data/pathway/metapathways2/libs/python_scripts/MetaPathways_run_pathologic.py", line 194, in main resultLines = pythonCyc.getReactionListLines() File "/data/pathway/metapathways2/libs/python_modules/utils/pathwaytoolsutils.py", line 341, in getReactionListLines my_base_pathways = self.call_func("all-pathways :all T") File "/data/pathway/metapathways2/libs/python_modules/utils/pathwaytoolsutils.py", line 159, in call_func self.send_query(self.wrap_query(function)) File "/data/pathway/metapathways2/libs/python_modules/utils/pathwaytoolsutils.py", line 82, in send_query if self.makeSocket(): File "/data/pathway/metapathways2/libs/python_modules/utils/pathwaytoolsutils.py", line 31, in makeSocket self.soc.connect("/tmp/ptools-socket" ) File "/usr/lib/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) error: [Errno 2] No such file or directory ERROR Failed to run extract pathways for test0 : INFO Kill any other PathwayTools instance running on the machine and try again Failed to Start pathway-tools INFO Extracting the reaction list from ePGDB test0 None INFO Entering the WTD calculations! INFO Getting MetaCyc Expected Taxonomic Range(s) Starting up pathway tools

..... Success! RUNNING RPKM_CALCULATION ..... Skipping [NO INPUT]! ***
INFO : FINISHED PROCESSING THE SAMPLES THE END

At the moment the error log is giving me:

STEP COMPUTE_RPKM

ERROR Missing input /data/pathway/project/input/reads

STEP COMPUTE_RPKM

ERROR Missing input no sam or fastq files to process [OPTIONAL]

Sorry for the long post. I hope it was of help. Many thanks again.

Giang

ariahahn commented 7 years ago

Those files you want there. Put the three back (its just sample files you want to run). Can you send me your config and param files?

Aria

On Wed, Oct 26, 2016 at 7:13 PM, GiangLeN notifications@github.com wrote:

Hi Aria,

Thank you for your reply. I do have ptools input now. So does that mean the file Metapathways_create_genbank.py is changeable with the Metapathways_create_genbank_ptinput_sequin.py file? In my directory (/data/pathway/project/ptools-local/pgdbs/user) there are 3 files: PGDB-counter.dat , PGDB-METADATA.ocelot and PGDB-METADATA.ocelot. I have moved them to a new directory for the test run.

Regarding rRNA databases, I downloaded GreenGenes and Silva fasta files and set these databases up together. However, metapathways only displays Silva result. For the test run here is the result (I deleted the RUNNING task to reduce the report length):

============== BEGIN RUN test0 19:24:53 BLOCK1 ================ FUNC_SEARCH:metacyc-v5-2011-10-21 SUCCESS FUNC_SEARCH:COG_2013-12-27 SUCCESS FUNC_SEARCH:CAZY_2014_09_04 SUCCESS

============== BEGIN RUN test0 19:24:53 BLOCK2 ================ COMPUTE_REFSCORES SUCCESS PARSE_FUNC_SEARCH:metacyc-v5-2011-10-21 SUCCESS PARSE_FUNC_SEARCH:COG_2013-12-27 SUCCESS PARSE_FUNC_SEARCH:CAZY_2014_09_04 SUCCESS SCAN_rRNA:SILVA_128_SSURef_Nr99_tax_silva SUCCESS SCAN_rRNA:Greengenes_13_15 SUCCESS SCAN_rRNA:SILVA_128_LSURef_tax_silva SUCCESS SCAN_tRNA SUCCESS ANNOTATE_ORFS SUCCESS GENBANK_FILE SUCCESS PATHOLOGIC_INPUT SUCCESS CREATE_ANNOT_REPORTS SUCCESS BUILD_PGDB RUNNING

The program would fail to build PGDB. I manually activated pathway-tools at the result folder (/data/pathway/project/output2/test0/ptools) as follow: /data/pathway/pathway-tools/pathway-tools -patho ./ -no-taxonomic-pruning -no-web-cel-overview -api After that the file pathologic.log file is created and I was able to get a green tick at build PGDB. Here is the log RUNNING PATHOLOGIC

RUNNING MetaPathways version 2.5.2 Fixing file /data/pathway/project/ptools-local/pgdbs/user/test0cyc/1.0/ input/organism.dat Starting up pathway tools

Traceback (most recent call last): File "/data/pathway/metapathways2/libs/python_scripts/ MetaPathways_run_pathologic.py", line 194, in main resultLines = pythonCyc.getReactionListLines() File "/data/pathway/metapathways2/libs/python_modules/utils/pathwaytoolsutils.py", line 341, in getReactionListLines my_base_pathways = self.call_func("all-pathways :all T") File "/data/pathway/metapathways2/libs/python_modules/utils/pathwaytoolsutils.py", line 159, in call_func self.send_query(self.wrap_query(function)) File "/data/pathway/metapathways2/libs/python_modules/utils/pathwaytoolsutils.py", line 82, in send_query if self.makeSocket(): File "/data/pathway/metapathways2/libs/python_modules/utils/pathwaytoolsutils.py", line 31, in makeSocket self.soc.connect("/tmp/ptools-socket" ) File "/usr/lib/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) error: [Errno 2] No such file or directory ERROR Failed to run extract pathways for test0 : INFO Kill any other PathwayTools instance running on the machine and try again Failed to Start pathway-tools INFO Extracting the reaction list from ePGDB test0 None INFO Entering the WTD calculations! INFO Getting MetaCyc Expected Taxonomic Range(s) Starting up pathway tools

..... Success! RUNNING RPKM_CALCULATION ..... Skipping [NO INPUT]!

INFO : FINISHED PROCESSING THE SAMPLES THE END

At the moment the error log is giving me:

STEP COMPUTE_RPKM

ERROR Missing input /data/pathway/project/input/reads

STEP COMPUTE_RPKM

ERROR Missing input no sam or fastq files to process [OPTIONAL]

Sorry for the long post. I hope it was of help. Many thanks again.

Giang

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hallamlab/metapathways2/issues/56#issuecomment-256528433, or mute the thread https://github.com/notifications/unsubscribe-auth/AI6RqJ-6IxpnftIiEumZ1nfyhD3InJO-ks5q4AjQgaJpZM4Dkolj .

Aria Hahn MSc PhD Candidate www.cupcakesandscience.com Hallam Laboratory http://www.cmde.science.ubc.ca/hallam/index.php

University of British Columbia Department of Microbiology & Immunology Life Sciences Centre 2350 Health Sciences Mall (Rm 2520) Vancouver, BC Canada V6T 1Z3 T: 604-827-4216 F: 604-822-6041 ariahahn@interchange.ubc.ca

GiangLeN commented 7 years ago

Are these the files you are looking for? template_param.txt template_config.txt

Giang

ariahahn commented 7 years ago

Hi Giang,

Please hold tight! I haven't forgotten and will be in touch soon.

Aria

On Oct 27, 2016 8:57 AM, "GiangLeN" notifications@github.com wrote:

Are these the files you are looking for? template_param.txt https://github.com/hallamlab/metapathways2/files/556373/template_param.txt template_config.txt https://github.com/hallamlab/metapathways2/files/556374/template_config.txt

Giang

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/hallamlab/metapathways2/issues/56#issuecomment-256687714, or mute the thread https://github.com/notifications/unsubscribe-auth/AI6RqIe_gxGn1Xf20RbISCUnk3jxxRUeks5q4MnYgaJpZM4Dkolj .

GiangLeN commented 7 years ago

Hi Aria,

Thank you for your help. No worries, I am still holding very tight. At the moment I am trying out version 2.5.2. So far, it also stucks at the build ePGDB step.

Cheer, Giang

GiangLeN commented 7 years ago

Hi Aria,

After trying out ver 2.5.2 here is what I found: It also stuck at the build ePGDB stage. When i try to rerun, I noticed that the sample folder in ptools-local/pgdbs/user is removed

I tried the command approach with this: python MetaPathways.py -i ../project/input/ -o ../project/out_2.5.2/ -c config/template_config.txt -p config/template_param.txt -v -r overlay -s seq1.fa

and it stuck at the Running Pathologic stage:

/data/pathway/metapathways2.5.2//libs/python_scripts/MetaPathways_run_pathologic.py --reactions /data/pathway/project/out_2.5.2/seq1/results//annotation_table//seq1.metacyc.orf.annots.txt --ptoolsExec /data/pathway/pathway-tools/pathway-tools -i /data/pathway/project/out_2.5.2/seq1/ptools// -p /data/pathway/project/ptools-local/pgdbs/user/seq1cyc -s seq1 --wtd --annotation-table /data/pathway/project/out_2.5.2/seq1/results//annotation_table//functional_and_taxonomic_table.txt --ncbi-tree /data/pathway/metapathways2.5.2/metapathways_dbs//ncbi_tree/ncbi_taxonomy_tree.txt --ncbi-megan-map /data/pathway/metapathways2.5.2/metapathways_dbs//ncbi_tree/ncbi.map --output-pwy-table /data/pathway/project/out_2.5.2/seq1/results//pgdb//seq1.pwy.txt

When i try to rerun this command I got this: Could not load some user defined module functions Make sure your typed 'source MetaPathwaysrc'

Traceback (most recent call last): File "/data/pathway/metapathways2.5.2//libs/python_scripts/MetaPathways_run_pathologic.py", line 10, in from libs.python_modules.taxonomy.LCAComputation import * ImportError: No module named libs.python_modules.taxonomy.LCAComputation

I am still unable to view MetaCyc result.

Many thanks, Giang

hallamlab commented 7 years ago

MetaCyc results can only be viewed upon generation of a ePGDB - a cyc file. We are soon releasing a new version which should address this issue!