Closed megaptera-helvetiae closed 4 years ago
Hi Laetitia,
(first of all I allowed myself to edit the formatting of your post so it's more readable; I hope you don't mind. I suggest you use the GitHub Markdown syntax e.g. enclosing commands and logs into triple backquote ```
so to clarify what's code and comments)
A general comment: you should not try to run downstream pipeline tasks when upstream pipeline tasks have failed, it's bound to fail too. Only task 04
is not required for subsequent tasks (as explained here). So when you've got an error, please try and fix the problem first.
About your error now!
the first error feeds from ls
are misleading e.g. in
ls: cannot access /scratch/clamchatka/Panta/test7/00.input_data/annotation/Ctena_galapagana_StHelenaBay_001_SYM/*.gbk: No such file or directory
are misleading; no files are missing, I think the script found an alternative file, in this case Ctena_mexicana_Hele014b.gbf
and could from there generate the modified GenBank file Ctena_mexicana_Hele014b.ptg.gbk
.
I suppressed that false error in commit 5c9db97.
the real problem seem to be the absence of modified GFF file Ctena_mexicana_Hele014b.ptg.gff
from the folder /scratch/clamchatka/Panta/test7/00.input_data/annotation/Ctena_mexicana_StHelenaBay_014_SYM2/
I think this issue, which somehow came unreported (but I should have fixed that too in commit 5c9db97), stems from an original failure in the Prokka
annotation for those genomes, as indicated by a non-empty errorsummary.val
file in that same folder.
So could you have a look at the content of that Prokka error log file and/or post it here please so I can to try and do a diagnostic?
Cheers,
Florent
ps: you seem to be running a dated version 9531df2 (2 months old); I made quite a few fixes to that part since, you should definitely updated the repository, cf. #26.
Hi Florent,
here is the output from errorsummary.val
total 56667
-rw-r--r--. 1 wilkins login 169 Nov 13 05:09 Ctena_mexicana_Hele011.ecn
-rw-r--r--. 1 wilkins login 885124 Nov 13 05:09 Ctena_mexicana_Hele011.err
-rw-r--r--. 1 wilkins login 1459440 Nov 13 05:09 Ctena_mexicana_Hele011.faa
-rw-r--r--. 1 wilkins login 4015993 Nov 13 05:09 Ctena_mexicana_Hele011.ffn
-rw-r--r--. 1 wilkins login 3028 Nov 13 05:09 Ctena_mexicana_Hele011.fixedproducts
-rw-r--r--. 1 wilkins login 4348700 Nov 13 05:07 Ctena_mexicana_Hele011.fna
-rw-r--r--. 1 wilkins login 4352210 Nov 13 05:09 Ctena_mexicana_Hele011.fsa
-rw-r--r--. 1 wilkins login 9445929 Nov 13 05:09 Ctena_mexicana_Hele011.gbf
-rw-r--r--. 1 wilkins login 6110321 Nov 13 05:09 Ctena_mexicana_Hele011.gff
-rw-r--r--. 1 wilkins login 56487 Nov 13 05:09 Ctena_mexicana_Hele011.log
-rw-r--r--. 1 wilkins login 9448919 Nov 13 05:09 Ctena_mexicana_Hele011.ptg.gbk
-rw-r--r--. 1 wilkins login 16329641 Nov 13 05:09 Ctena_mexicana_Hele011.sqn
-rw-r--r--. 1 wilkins login 1160610 Nov 13 05:09 Ctena_mexicana_Hele011.tbl
-rw-r--r--. 1 wilkins login 402003 Nov 13 05:09 Ctena_mexicana_Hele011.tsv
-rw-r--r--. 1 wilkins login 108 Nov 13 05:09 Ctena_mexicana_Hele011.txt
-rw-r--r--. 1 wilkins login 3311 Nov 13 05:09 Ctena_mexicana_Hele011.val
-rw-r--r--. 1 wilkins login 132 Nov 13 05:09 errorsummary.val
[wilkins@gorilla Ctena_mexicana_StHelenaBay_011_SYM]$ cat errorsummary.val
1 ERROR: SEQ_FEAT.BadProteinName
3 WARNING: SEQ_FEAT.BadEcNumberValue
9 WARNING: SEQ_FEAT.ProteinNameEndsInBracket```
So here is a little update.
I started from scratch. Created a new repository using the following commands:
git clone https://github.com/flass/pantagruel.git
Cloning into 'pantagruel'...
remote: Enumerating objects: 41, done.
remote: Counting objects: 100% (41/41), done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 7243 (delta 21), reused 21 (delta 7), pack-reused 7202
Receiving objects: 100% (7243/7243), 6.98 MiB | 7.96 MiB/s, done.
Resolving deltas: 100% (5595/5595), done.
Checking out files: 100% (181/181), done.
[wilkins@gorilla scripts]$ git pull
Already up to date.
Then, I changed the path in my environment file that it would find the scripts at the new locations.
#!/bin/bash
## Pantagruel database 'test8'
## built with Pantagruel version '9531df2e57fd032f1ff8e11b79091953833f978e'; source code available at 'https://github.com/flass/pantagruel'
# the init command with which the config file was created
ptginitcmd='pantagruel -d test8 -r /scratch/clamchatka/Panta/ -a /scratch/clamchatka/Panta/user_genomes/ -T /scratch/clamchatka/Panta/NCBI/Taxonomy_2019-11-12/ init'
# location (folder) of Pantagruel software that was used
export ptgrepo='/apps/pantagruel/9531df2e57fd032f1ff8e11b79091953833f978e'
# derive paths to Pantagruel scripts and Python modules
export ptgscripts='/scratch/clamchatka/Panta/pantagruel/scripts'
export PYTHONPATH='/scratch/clamchatka/Panta/pantagruel/python_libs'
# database parameters (primary variables)
export ptgroot='/scratch/clamchatka/Panta' # root folder where to build the database
export ptgdbname='test8' # name of dataabse
export ptgversinit='9531df2e57fd032f1ff8e11b79091953833f978e' # current version of Pantagruel software
export myemail='undisclosed' # user identity (better use e-amil address)
export famprefix='PANTAG' # gene family prefix
export ncbitax='/scratch/clamchatka/Panta/NCBI/Taxonomy_2019-11-12' # folder of up-to-date NCBI Taxonomy database
export ncbiass='' # folder of RefSeq genomes to include in the study
export listncbiass='' # list of accessions of RefSeq genomes to include in the study
export customassemb='/scratch/clamchatka/Panta/user_genomes' # folder of custom genome assemblies to include in the study
export refass='' # folder of reference (RefSeq) genomes only to use as reference for the annotation of custom genome assemblies
export listrefass='' # list of accessions of reference (RefSeq) genomes only to use as reference for the annotation of custom genome assemblies
export coreseqtype='cds' # either 'cds' or 'protein'
export pseudocoremingenomes= # the minimum number of genomes in which a gene family should be present to be included in the pseudo-core genome gene set
export userreftree='' # possible user-provided reference tree
export poplgthresh='default' # parameter to define populations of genomes in the reference tree (stem branch length threshold, default value depends on coreseqtype)
export poplgleafmul='1.5' # parameter to define populations of genomes in the reference tree (multiplier to the former in case it is a leaf, default 1.5)
export popbsthresh='80' # parameter to define populations of genomes in the reference tree (stem branch support threshold, default 80)
export rootingmethod='treebalance'
export chaintype='fullgenetree' # whether gene trees will be collapsed ('collapsed', if -c option enabled) or not ('fullgenetree', default)
export genefamlist='' # list of gene families for which computation of gene trees and all subsequent analyses will be restricted (default: no restriction)
# non-default parameters for gene trees collapsing derived from -C option value (passed to init script via ${collapseCladeParams}):
export cladesupp=70 # - clade criterion trheshold (int)
export subcladesupp=35 # - wihtin-clade criterion trheshold (int)
export criterion='bs' # - criterion (branch support: 'bs', branch length 'lg')
export withinfun='median' # - aggregate function for testing within the clade ('min', 'max', 'mean', 'median')
export hpcremoteptgroot='none' # if not empty nor 'none', will use this server address to send data and scripts to run heavy computions there
# other parameters have default values defined in the generic source file environ_pantagruel_defaults.sh
source ${ptgscripts}/pipeline/environ_pantagruel_defaults.sh
# these defalts can be overriden by uncommenting the relevant line below and editing the variable's value
# default values are:
# Prokka annotation parameters (only relevant if custom genome assemblies are provided):
#~ export assembler="somesoftware"
#~ export seqcentre="somewhere"
#~ export refgenus="Reference"
# species tree inference parameters
#~ export ncorebootstrap=200
# gene tree inference parameters
#~ export mainresulttag='rootedTree'
# gene trees collapsing DEFAULT values (used when -C option is NOT present in init call)
#~ export cladesuppdef=70
#~ export subcladesuppdef=35
#~ export criteriondef='bs'
#~ export withinfundef='median'
# gene tree/species tree reconciliation inference parameters
#~ export ALEalgo='ALEml'
#~ export recsamplesize=1000
# gene tree/species tree reconciliation parsing parameters for co-evolution analysis
#~ export evtypeparse='ST'
#~ export minevfreqparse=0.1
#~ export minevfreqmatch=0.5
#~ export minjoinevfreqmatch=1.0
#~ export maxreftreeheight=0.25
# secondary vars are defined based on the above
source ${ptgscripts}/pipeline/environ_pantagruel_secondaryvars.sh
# load shared functions
source ${ptgscripts}/pipeline/pantagruel_pipeline_functions.sh
Particularly this: export ptgscripts='/scratch/clamchatka/Panta/pantagruel/scripts' export PYTHONPATH='/scratch/clamchatka/Panta/pantagruel/python_libs'
Then, I tried to run prokka again. This time it got the error much earlier.
pantagruel -i /scratch/clamchatka/Panta/test8/environ_pantagruel_test8.sh fetch -F
This is Pantagruel pipeline version 9531df2e57fd032f1ff8e11b79091953833f978e using source code from repository '/apps/pantagruel/9531df2e57fd032f1ff8e11b79091953833f978e'
# will run tasks: 0
[2019-11-14 04:40:52] Pantagrel pipeline task 0: fetch public genome data from NCBI sequence databases and annotate private genomes.
Task folder '/scratch/clamchatka/Panta/test8/00.input_data' already exists; FORCE mode is on: ERASE and recreate the folder to write new result in its place
[2019-11-14 04:40:52] extract assembly data from folder '/scratch/clamchatka/Panta/user_genomes'
found 14 contig files (raw genome assemblies) in /scratch/clamchatka/Panta/user_genomes/contigs/
[2019-11-14 04:40:53] Ctena_galapagana_StHelenaBay_001_SYM
will annotate contigs in '/scratch/clamchatka/Panta/user_genomes/contigs/Ctena_galapagana_StHelenaBay_001_SYM.fasta'
[2019-11-14 04:40:53]
### assembly: Ctena_galapagana_StHelenaBay_001_SYM; contig files from: /scratch/clamchatka/Panta/user_genomes/contigs/Ctena_galapagana_StHelenaBay_001_SYM.fasta
running Prokka...
done.
[2019-11-14 04:42:29]
fix annotation to integrate region information into GFF files
fix annotation to integrate taxid information into GBK files
ls: cannot access /scratch/clamchatka/Panta/test8/00.input_data/annotation/Ctena_galapagana_StHelenaBay_001_SYM/*.gbk: No such file or directory
Traceback (most recent call last):
File "/scratch/clamchatka/Panta/pantagruel/scripts/add_taxid_feature2prokkaGBK.py", line 19, in <module>
strain = lsp[strainfield]
IndexError: list index out of range
ERROR: something went wrong when modifying the GenBank flat file /scratch/clamchatka/Panta/test8/00.input_data/annotation/Ctena_galapagana_StHelenaBay_001_SYM/Ctena_mexicana_Hele001.gbf
ERROR: Pantagrel pipeline task 0: failed.
The error is still the same:
cat errorsummary.val
1 ERROR: SEQ_FEAT.BadProteinName
3 WARNING: SEQ_FEAT.BadEcNumberValue
7 WARNING: SEQ_FEAT.ProteinNameEndsInBracket
Hi Laetitia,
I am confused why you would get the error earlier, as it seems you are still running on the same version (9531df2)!
I believe the pantagruel
command links to the repository that was once installed by your admins. indeed the variable locating the active repository is defined as :
ptgrepo='/apps/pantagruel/9531df2e57fd032f1ff8e11b79091953833f978e'
Instead, you should use the explicit path to the command:
/where/you/cloned/the/last/version/pantagruel/pantagruel
For instance, you can just refresh the config file using:
/where/you/cloned/the/last/version/pantagruel/pantagruel -i /scratch/clamchatka/Panta/test8/environ_pantagruel_test8.sh --refresh init
This should automatically set up the $ptgscripts
and $PYTHONPATH
correctly (but also importantly, the variable $ptgrepo
)
But before doing that, I strongly suggest you run those additional commands to set up the submodule tree2
(cf. install_dependencies.sh
script), otherwise it will soon cause problems:
cd /where/you/cloned/the/last/version/pantagruel/
git submodule init
git submodule update
then for every routine update, please make sure you use:
git pull
git submodule update
So once you've done that we should see a bit clearer, as for the moment I have the impression you run a mixture of old pipeline scripts with recent python modules (as your $PYTHONPATH
point s to the the recent repository version).
Now regarding the problem underlying your errors, my first guess would be that you may not have set (correctly) the /scratch/clamchatka/Panta/user_genomes/strain_info_test8.txt
file. Can I have a glimpse of how it looks please?
Also I suspect that there would be further information on the failure of prokka
in the file Ctena_mexicana_Hele011.val
, as it is not part of the regular bunch of output files
To initialize I run:
pantagruel -d test9 -r /scratch/clamchatka/Panta/ -a /scratch/clamchatka/Panta/user_genomes/ -T /scratch/clamchatka/Panta/NCBI/Taxonomy_2019-11-12/ init
I did setup the repository and update all the newest scripts and submodules as you suggested. This is now in /scratch/clamchatka/Panta/pantagruel.
Then I refresh the config_file:
/scratch/clamchatka/Panta/pantagruel/pantagruel -i /scratch/clamchatka/Panta/test9/environ_pantagruel_test9.sh --refresh init
Then I go into the config_file and check but it did not update the paths to my updated scripts!
See yourself:
#!/bin/bash
## Pantagruel database 'test9'
## built with Pantagruel version '9531df2e57fd032f1ff8e11b79091953833f978e'; source code available at 'https://github.com/flass/pantagruel'
# the init command with which the config file was created
ptginitcmd='pantagruel -d test9 -r /scratch/clamchatka/Panta/ -a /scratch/clamchatka/Panta/user_genomes/ -T /scratch/clamchatka/Panta/NCBI/Taxonomy_2019-11-12/ init'
# location (folder) of Pantagruel software that was used
export ptgrepo='/apps/pantagruel/9531df2e57fd032f1ff8e11b79091953833f978e'
# derive paths to Pantagruel scripts and Python modules
export ptgscripts=${ptgrepo}/scripts
export PYTHONPATH=${ptgrepo}/python_libs
# database parameters (primary variables)
export ptgroot='/scratch/clamchatka/Panta' # root folder where to build the database
export ptgdbname='test9' # name of dataabse
export ptgversinit='9531df2e57fd032f1ff8e11b79091953833f978e' # current version of Pantagruel software
export myemail='undisclosed' # user identity (better use e-amil address)
export famprefix='PANTAG' # gene family prefix
export ncbitax='/scratch/clamchatka/Panta/NCBI/Taxonomy_2019-11-12' # folder of up-to-date NCBI Taxonomy database
export ncbiass='' # folder of RefSeq genomes to include in the study
export listncbiass='' # list of accessions of RefSeq genomes to include in the study
export customassemb='/scratch/clamchatka/Panta/user_genomes' # folder of custom genome assemblies to include in the study
export refass='' # folder of reference (RefSeq) genomes only to use as reference for the annotation of custom genome assemblies
export listrefass='' # list of accessions of reference (RefSeq) genomes only to use as reference for the annotation of custom genome assemblies
export coreseqtype='cds' # either 'cds' or 'protein'
export pseudocoremingenomes= # the minimum number of genomes in which a gene family should be present to be included in the pseudo-core genome gene set
export userreftree='' # possible user-provided reference tree
export poplgthresh='default' # parameter to define populations of genomes in the reference tree (stem branch length threshold, default value depends on coreseqtype)
export poplgleafmul='1.5' # parameter to define populations of genomes in the reference tree (multiplier to the former in case it is a leaf, default 1.5)
export popbsthresh='80' # parameter to define populations of genomes in the reference tree (stem branch support threshold, default 80)
export rootingmethod='treebalance'
export chaintype='fullgenetree' # whether gene trees will be collapsed ('collapsed', if -c option enabled) or not ('fullgenetree', default)
export genefamlist='' # list of gene families for which computation of gene trees and all subsequent analyses will be restricted (default: no restriction)
# non-default parameters for gene trees collapsing derived from -C option value (passed to init script via ${collapseCladeParams}):
export cladesupp=70 # - clade criterion trheshold (int)
export subcladesupp=35 # - wihtin-clade criterion trheshold (int)
export criterion='bs' # - criterion (branch support: 'bs', branch length 'lg')
export withinfun='median' # - aggregate function for testing within the clade ('min', 'max', 'mean', 'median')
export hpcremoteptgroot='none' # if not empty nor 'none', will use this server address to send data and scripts to run heavy computions there
# other parameters have default values defined in the generic source file environ_pantagruel_defaults.sh
source ${ptgscripts}/pipeline/environ_pantagruel_defaults.sh
# these defalts can be overriden by uncommenting the relevant line below and editing the variable's value
# default values are:
# Prokka annotation parameters (only relevant if custom genome assemblies are provided):
#~ export assembler="somesoftware"
#~ export seqcentre="somewhere"
#~ export refgenus="Reference"
# species tree inference parameters
#~ export ncorebootstrap=200
# gene tree inference parameters
#~ export mainresulttag='rootedTree'
# gene trees collapsing DEFAULT values (used when -C option is NOT present in init call)
#~ export cladesuppdef=70
#~ export subcladesuppdef=35
#~ export criteriondef='bs'
#~ export withinfundef='median'
# gene tree/species tree reconciliation inference parameters
#~ export ALEalgo='ALEml'
#~ export recsamplesize=1000
# gene tree/species tree reconciliation parsing parameters for co-evolution analysis
#~ export evtypeparse='ST'
#~ export minevfreqparse=0.1
#~ export minevfreqmatch=0.5
#~ export minjoinevfreqmatch=1.0
#~ export maxreftreeheight=0.25
# secondary vars are defined based on the above
source ${ptgscripts}/pipeline/environ_pantagruel_secondaryvars.sh
# load shared functions
source ${ptgscripts}/pipeline/pantagruel_pipeline_functions.sh
Should I now change the paths to ptgrepo, scripts and python libraries by hand???
I did this here to test:
#!/bin/bash
## Pantagruel database 'test9'
## built with Pantagruel version '9531df2e57fd032f1ff8e11b79091953833f978e'; source code available at 'https://github.com/flass/pantagruel'
# the init command with which the config file was created
ptginitcmd='pantagruel -d test9 -r /scratch/clamchatka/Panta/ -a /scratch/clamchatka/Panta/user_genomes/ -T /scratch/clamchatka/Panta/NCBI/Taxonomy_2019-11-12/ init'
# location (folder) of Pantagruel software that was used
export ptgrepo='/scratch/clamchatka/Panta/pantagruel'
# derive paths to Pantagruel scripts and Python modules
export ptgscripts='/scratch/clamchatka/Panta/pantagruel/scripts'
export PYTHONPATH='/scratch/clamchatka/Panta/pantagruel/python_libs'
# database parameters (primary variables)
export ptgroot='/scratch/clamchatka/Panta' # root folder where to build the database
export ptgdbname='test9' # name of dataabse
export ptgversinit='4867c048788ba7ec92dfd5ae9148d0349411151c' # current version of Pantagruel software
export myemail='undisclosed' # user identity (better use e-amil address)
export famprefix='PANTAG' # gene family prefix
export ncbitax='/scratch/clamchatka/Panta/NCBI/Taxonomy_2019-11-12' # folder of up-to-date NCBI Taxonomy database
export ncbiass='' # folder of RefSeq genomes to include in the study
export listncbiass='' # list of accessions of RefSeq genomes to include in the study
export customassemb='/scratch/clamchatka/Panta/user_genomes' # folder of custom genome assemblies to include in the study
export refass='' # folder of reference (RefSeq) genomes only to use as reference for the annotation of custom genome assemblies
export listrefass='' # list of accessions of reference (RefSeq) genomes only to use as reference for the annotation of custom genome assemblies
export coreseqtype='cds' # either 'cds' or 'protein'
export pseudocoremingenomes= # the minimum number of genomes in which a gene family should be present to be included in the pseudo-core genome gene set
export userreftree='' # possible user-provided reference tree
export poplgthresh='default' # parameter to define populations of genomes in the reference tree (stem branch length threshold, default value depends on coreseqtype)
export poplgleafmul='1.5' # parameter to define populations of genomes in the reference tree (multiplier to the former in case it is a leaf, default 1.5)
export popbsthresh='80' # parameter to define populations of genomes in the reference tree (stem branch support threshold, default 80)
export rootingmethod='treebalance'
export chaintype='fullgenetree' # whether gene trees will be collapsed ('collapsed', if -c option enabled) or not ('fullgenetree', default)
export genefamlist='' # list of gene families for which computation of gene trees and all subsequent analyses will be restricted (default: no restriction)
# non-default parameters for gene trees collapsing derived from -C option value (passed to init script via ${collapseCladeParams}):
export cladesupp=70 # - clade criterion trheshold (int)
export subcladesupp=35 # - wihtin-clade criterion trheshold (int)
export criterion='bs' # - criterion (branch support: 'bs', branch length 'lg')
export withinfun='median' # - aggregate function for testing within the clade ('min', 'max', 'mean', 'median')
export hpcremoteptgroot='none' # if not empty nor 'none', will use this server address to send data and scripts to run heavy computions there
# other parameters have default values defined in the generic source file environ_pantagruel_defaults.sh
source ${ptgscripts}/pipeline/environ_pantagruel_defaults.sh
# these defalts can be overriden by uncommenting the relevant line below and editing the variable's value
# default values are:
# Prokka annotation parameters (only relevant if custom genome assemblies are provided):
#~ export assembler="somesoftware"
#~ export seqcentre="somewhere"
#~ export refgenus="Reference"
# species tree inference parameters
#~ export ncorebootstrap=200
# gene tree inference parameters
#~ export mainresulttag='rootedTree'
# gene trees collapsing DEFAULT values (used when -C option is NOT present in init call)
#~ export cladesuppdef=70
#~ export subcladesuppdef=35
#~ export criteriondef='bs'
#~ export withinfundef='median'
# gene tree/species tree reconciliation inference parameters
#~ export ALEalgo='ALEml'
#~ export recsamplesize=1000
# gene tree/species tree reconciliation parsing parameters for co-evolution analysis
#~ export evtypeparse='ST'
#~ export minevfreqparse=0.1
#~ export minevfreqmatch=0.5
#~ export minjoinevfreqmatch=1.0
#~ export maxreftreeheight=0.25
# secondary vars are defined based on the above
source ${ptgscripts}/pipeline/environ_pantagruel_secondaryvars.sh
# load shared functions
source ${ptgscripts}/pipeline/pantagruel_pipeline_functions.sh
You see, I even changed the version manually!
Then I run Prokka:
pantagruel -i /scratch/clamchatka/Panta/test9/environ_pantagruel_test9.sh fetch
And I get this error:
This is Pantagruel pipeline version 9531df2e57fd032f1ff8e11b79091953833f978e using source code from repository '/apps/pantagruel/9531df2e57fd032f1ff8e11b79091953833f978e'
# will run tasks: 0
[2019-11-14 20:27:45] Pantagrel pipeline task 0: fetch public genome data from NCBI sequence databases and annotate private genomes.
ERROR: the current version of pantagruel (commit 9531df2) is different from the one used to generate the config file '/scratch/clamchatka/Panta/test9/environ_pantagruel_test9.sh' (commit 4867c04).
Please regenerate the config file with `pantagruel init` to ensure compatibility; for the same parameters to be set, just run the same command with same options as previously.
ERROR: Pantagrel pipeline task 0: failed.
So I refresh my config file:
/scratch/clamchatka/Panta/pantagruel/pantagruel -i /scratch/clamchatka/Panta/test9/environ_pantagruel_test9.sh --refresh init
Note, this put it back to the older version of pantagruel (export ptgrepo='/apps/pantagruel/9531df2e57fd032f1ff8e11b79091953833f978e'
).
Here you see what happened to the config file:
#!/bin/bash
## Pantagruel database 'test9'
## built with Pantagruel version '9531df2e57fd032f1ff8e11b79091953833f978e'; source code available at 'https://github.com/flass/pantagruel'
# the init command with which the config file was created
ptginitcmd='pantagruel -d test9 -r /scratch/clamchatka/Panta/ -a /scratch/clamchatka/Panta/user_genomes/ -T /scratch/clamchatka/Panta/NCBI/Taxonomy_2019-11-12/ init'
# location (folder) of Pantagruel software that was used
export ptgrepo='/apps/pantagruel/9531df2e57fd032f1ff8e11b79091953833f978e'
# derive paths to Pantagruel scripts and Python modules
export ptgscripts=${ptgrepo}/scripts
export PYTHONPATH=${ptgrepo}/python_libs
# database parameters (primary variables)
export ptgroot='/scratch/clamchatka/Panta' # root folder where to build the database
export ptgdbname='test9' # name of dataabse
export ptgversinit='9531df2e57fd032f1ff8e11b79091953833f978e' # current version of Pantagruel software
export myemail='undisclosed' # user identity (better use e-amil address)
export famprefix='PANTAG' # gene family prefix
export ncbitax='/scratch/clamchatka/Panta/NCBI/Taxonomy_2019-11-12' # folder of up-to-date NCBI Taxonomy database
export ncbiass='' # folder of RefSeq genomes to include in the study
export listncbiass='' # list of accessions of RefSeq genomes to include in the study
export customassemb='/scratch/clamchatka/Panta/user_genomes' # folder of custom genome assemblies to include in the study
export refass='' # folder of reference (RefSeq) genomes only to use as reference for the annotation of custom genome assemblies
export listrefass='' # list of accessions of reference (RefSeq) genomes only to use as reference for the annotation of custom genome assemblies
export coreseqtype='cds' # either 'cds' or 'protein'
export pseudocoremingenomes= # the minimum number of genomes in which a gene family should be present to be included in the pseudo-core genome gene set
export userreftree='' # possible user-provided reference tree
export poplgthresh='default' # parameter to define populations of genomes in the reference tree (stem branch length threshold, default value depends on coreseqtype)
export poplgleafmul='1.5' # parameter to define populations of genomes in the reference tree (multiplier to the former in case it is a leaf, default 1.5)
export popbsthresh='80' # parameter to define populations of genomes in the reference tree (stem branch support threshold, default 80)
export rootingmethod='treebalance'
export chaintype='fullgenetree' # whether gene trees will be collapsed ('collapsed', if -c option enabled) or not ('fullgenetree', default)
export genefamlist='' # list of gene families for which computation of gene trees and all subsequent analyses will be restricted (default: no restriction)
# non-default parameters for gene trees collapsing derived from -C option value (passed to init script via ${collapseCladeParams}):
export cladesupp=70 # - clade criterion trheshold (int)
export subcladesupp=35 # - wihtin-clade criterion trheshold (int)
export criterion='bs' # - criterion (branch support: 'bs', branch length 'lg')
export withinfun='median' # - aggregate function for testing within the clade ('min', 'max', 'mean', 'median')
export hpcremoteptgroot='none' # if not empty nor 'none', will use this server address to send data and scripts to run heavy computions there
# other parameters have default values defined in the generic source file environ_pantagruel_defaults.sh
source ${ptgscripts}/pipeline/environ_pantagruel_defaults.sh
# these defalts can be overriden by uncommenting the relevant line below and editing the variable's value
# default values are:
# Prokka annotation parameters (only relevant if custom genome assemblies are provided):
#~ export assembler="somesoftware"
#~ export seqcentre="somewhere"
#~ export refgenus="Reference"
# species tree inference parameters
#~ export ncorebootstrap=200
# gene tree inference parameters
#~ export mainresulttag='rootedTree'
# gene trees collapsing DEFAULT values (used when -C option is NOT present in init call)
#~ export cladesuppdef=70
#~ export subcladesuppdef=35
#~ export criteriondef='bs'
#~ export withinfundef='median'
# gene tree/species tree reconciliation inference parameters
#~ export ALEalgo='ALEml'
#~ export recsamplesize=1000
# gene tree/species tree reconciliation parsing parameters for co-evolution analysis
#~ export evtypeparse='ST'
#~ export minevfreqparse=0.1
#~ export minevfreqmatch=0.5
#~ export minjoinevfreqmatch=1.0
#~ export maxreftreeheight=0.25
# secondary vars are defined based on the above
source ${ptgscripts}/pipeline/environ_pantagruel_secondaryvars.sh
# load shared functions
source ${ptgscripts}/pipeline/pantagruel_pipeline_functions.sh
My strain info file looks like this:
[wilkins@gorilla user_genomes]$ cat strain_infos_test9.txt
assembly_id genus species strain taxid locus_tax_prefix
Ctena_galapagana_StHelenaBay_001_SYM Ctena mexicana Hele001 1655433 CtegalHel001
Ctena_imbricatula_STRI_051_SYM Ctena mexicana STRI051 1655433 CteimSTRI051
Ctena_imbricatula_STRI_052_SYM Ctena mexicana STRI052 1655433 CteimSTRI052
Ctena_imbricatula_STRI_065_SYM Ctena mexicana STRI065 1655433 CteimSTRI065
Ctena_imbricatula_STRI_068_SYM Ctena mexicana STRI068 1655433 CteimSTRI068
Ctena_imbricatula_STRI_070_SYM Ctena mexicana STRI070 1655433 CteimSTRI070
Ctena_imbricatula_STRI_073_SYM Ctena mexicana STRI073 1655433 CteimSTRI073
Ctena_imbricatula_STRI_074_SYM Ctena mexicana STRI074 1655433 CteimSTRI074
Ctena_imbricatula_STRI_094_SYM Ctena mexicana STRI094 1655433 CteimSTRI094
Ctena_mexicana_StHelenaBay_011_SYM Ctena mexicana Hele011 1655433 CtegalHel011
Ctena_mexicana_StHelenaBay_012_SYM Ctena mexicana Hele012 1655433 CtegalHel012
Ctena_mexicana_StHelenaBay_013_SYM Ctena mexicana Hele013 1655433 CtegalHel013
Ctena_mexicana_StHelenaBay_014_SYM2 Ctena mexicana Hele014b 1655433 CtegalHel014b
Ctena_mexicana_StHelenaBay_014_SYM Ctena mexicana Hele014 1655433 CtegalHel014
So I try to run prokka now:
pantagruel -i /scratch/clamchatka/Panta/test9/environ_pantagruel_test9.sh fetch
We are still getting the same error:
running Prokka...
done.
[2019-11-14 20:46:46]
fix annotation to integrate region information into GFF files
fix annotation to integrate taxid information into GBK files
ls: cannot access /scratch/clamchatka/Panta/test9/00.input_data/annotation/Ctena_imbricatula_STRI_070_SYM/*.gbk: No such file or directory
done.
[2019-11-14 21:04:33]
fix annotation to integrate region information into GFF files
fix annotation to integrate taxid information into GBK files
ls: cannot access /scratch/clamchatka/Panta/test9/00.input_data/annotation/Ctena_mexicana_StHelenaBay_014_SYM/*.gbk: No such file or directory
done.
will create GenBank-like assembly folders for user-provided genomes
gzip: /scratch/clamchatka/Panta/test9/00.input_data/annotation/Ctena_galapagana_StHelenaBay_001_SYM/ is a directory -- ignored
gzip: /scratch/clamchatka/Panta/test9/00.input_data/annotation/Ctena_imbricatula_STRI_051_SYM/ is a directory -- ignored
gzip: /scratch/clamchatka/Panta/test9/00.input_data/annotation/Ctena_imbricatula_STRI_052_SYM/ is a directory -- ignored
gzip: /scratch/clamchatka/Panta/test9/00.input_data/annotation/Ctena_imbricatula_STRI_065_SYM/ is a directory -- ignored
gzip: /scratch/clamchatka/Panta/test9/00.input_data/annotation/Ctena_imbricatula_STRI_068_SYM/ is a directory -- ignored
gzip: /scratch/clamchatka/Panta/test9/00.input_data/annotation/Ctena_imbricatula_STRI_070_SYM/ is a directory -- ignored
gzip: /scratch/clamchatka/Panta/test9/00.input_data/annotation/Ctena_imbricatula_STRI_073_SYM/ is a directory -- ignored
gzip: /scratch/clamchatka/Panta/test9/00.input_data/annotation/Ctena_imbricatula_STRI_074_SYM/ is a directory -- ignored
gzip: /scratch/clamchatka/Panta/test9/00.input_data/annotation/Ctena_imbricatula_STRI_094_SYM/ is a directory -- ignored
gzip: /scratch/clamchatka/Panta/test9/00.input_data/annotation/Ctena_mexicana_StHelenaBay_011_SYM/ is a directory -- ignored
gzip: /scratch/clamchatka/Panta/test9/00.input_data/annotation/Ctena_mexicana_StHelenaBay_012_SYM/ is a directory -- ignored
gzip: /scratch/clamchatka/Panta/test9/00.input_data/annotation/Ctena_mexicana_StHelenaBay_013_SYM/ is a directory -- ignored
gzip: /scratch/clamchatka/Panta/test9/00.input_data/annotation/Ctena_mexicana_StHelenaBay_014_SYM/ is a directory -- ignored
gzip: /scratch/clamchatka/Panta/test9/00.input_data/annotation/Ctena_mexicana_StHelenaBay_014_SYM2/ is a directory -- ignored
Traceback (most recent call last):
File "/apps/pantagruel/9531df2e57fd032f1ff8e11b79091953833f978e/scripts/extract_metadata_from_gbff.py", line 366, in <module>
main(nfldirassemb, dirassemblyinfo, output, defspename, nfdhandmetaraw, nfdhandmetacur, nfdhanddbxref)
File "/apps/pantagruel/9531df2e57fd032f1ff8e11b79091953833f978e/scripts/extract_metadata_from_gbff.py", line 72, in main
lassemb = [parse_assembly_name(assembname, reass=reass) for assembname in lassembname]
File "/apps/pantagruel/9531df2e57fd032f1ff8e11b79091953833f978e/scripts/extract_metadata_from_gbff.py", line 60, in parse_assembly_name
geass = seass.groups()
AttributeError: 'NoneType' object has no attribute 'groups'
[2019-11-14 21:06:14]
Pantagrel pipeline task 0: complete.
Error file looks like this:
[wilkins@gorilla Ctena_mexicana_StHelenaBay_011_SYM]$ cat Ctena_mexicana_Hele011.val
WARNING: valid [SEQ_FEAT.ProteinNameEndsInBracket] Protein name ends with bracket and may contain organism name FEATURE: Prot: Superoxide dismutase [Fe] [gnl|somewhere|CtegalHel011_00826:1-193] [gnl|somewhere|CtegalHel011_00826: raw, aa len= 193]
WARNING: valid [SEQ_FEAT.ProteinNameEndsInBracket] Protein name ends with bracket and may contain organism name FEATURE: Prot: Nicotinate-nucleotide pyrophosphorylase [carboxylating] [gnl|somewhere|CtegalHel011_00899:1-280] [gnl|somewhere|CtegalHel011_00899: raw, aa len= 280]
WARNING: valid [SEQ_FEAT.ProteinNameEndsInBracket] Protein name ends with bracket and may contain organism name FEATURE: Prot: GMP synthase [glutamine-hydrolyzing] [gnl|somewhere|CtegalHel011_01114:1-239] [gnl|somewhere|CtegalHel011_01114: raw, aa len= 239]
ERROR: valid [SEQ_FEAT.BadProteinName] Unknown or hypothetical protein should not have EC number FEATURE: Prot: hypothetical protein [gnl|somewhere|CtegalHel011_01239:1-321] [gnl|somewhere|CtegalHel011_01239: raw, aa len= 321]
WARNING: valid [SEQ_FEAT.ProteinNameEndsInBracket] Protein name ends with bracket and may contain organism name FEATURE: Prot: GMP synthase [glutamine-hydrolyzing] [gnl|somewhere|CtegalHel011_01596:1-194] [gnl|somewhere|CtegalHel011_01596: raw, aa len= 194]
WARNING: valid [SEQ_FEAT.ProteinNameEndsInBracket] Protein name ends with bracket and may contain organism name FEATURE: Prot: GMP synthase [glutamine-hydrolyzing] [gnl|somewhere|CtegalHel011_01905:1-239] [gnl|somewhere|CtegalHel011_01905: raw, aa len= 239]
WARNING: valid [SEQ_FEAT.BadEcNumberValue] 2.4.1.345 is not a legal value for qualifier EC_number FEATURE: Prot: Phosphatidyl-myo-inositol mannosyltransferase [gnl|somewhere|CtegalHel011_02328:1-404] [gnl|somewhere|CtegalHel011_02328: raw, aa len= 404]
WARNING: valid [SEQ_FEAT.BadEcNumberValue] 1.17.1.9 is not a legal value for qualifier EC_number FEATURE: Prot: Formate dehydrogenase H [gnl|somewhere|CtegalHel011_02586:1-915] [gnl|somewhere|CtegalHel011_02586: raw, aa len= 915]
WARNING: valid [SEQ_FEAT.ProteinNameEndsInBracket] Protein name ends with bracket and may contain organism name FEATURE: Prot: Phosphoenolpyruvate carboxykinase [GTP] [gnl|somewhere|CtegalHel011_03044:1-619] [gnl|somewhere|CtegalHel011_03044: raw, aa len= 619]
WARNING: valid [SEQ_FEAT.BadEcNumberValue] 1.17.1.9 is not a legal value for qualifier EC_number FEATURE: Prot: Formate dehydrogenase-O major subunit [gnl|somewhere|CtegalHel011_03055:1-964] [gnl|somewhere|CtegalHel011_03055: raw, aa len= 964]
WARNING: valid [SEQ_FEAT.ProteinNameEndsInBracket] Protein name ends with bracket and may contain organism name FEATURE: Prot: GMP synthase [glutamine-hydrolyzing] [gnl|somewhere|CtegalHel011_03205:1-70] [gnl|somewhere|CtegalHel011_03205: raw, aa len= 70]
WARNING: valid [SEQ_FEAT.ProteinNameEndsInBracket] Protein name ends with bracket and may contain organism name FEATURE: Prot: Glutamine--fructose-6-phosphate aminotransferase [isomerizing] [gnl|somewhere|CtegalHel011_03459:1-608] [gnl|somewhere|CtegalHel011_03459: raw, aa len= 608]
WARNING: valid [SEQ_FEAT.ProteinNameEndsInBracket] Protein name ends with bracket and may contain organism name FEATURE: Prot: GMP synthase [glutamine-hydrolyzing] [gnl|somewhere|CtegalHel011_03858:1-158] [gnl|somewhere|CtegalHel011_03858: raw, aa len= 158]
Can you help me update the repo and link it to the config file? I think this will save many issues I am having.
it is annoying that using pantagruel --refresh init
with your local install i.e.
/scratch/clamchatka/Panta/pantagruel/pantagruel -i /scratch/clamchatka/Panta/test9/environ_pantagruel_test9.sh --refresh init
you still end up with reference to the old version of the repo.
This might have to do with you having a system-wide installation of pantagruel
that somehow takes precedence on your local one in determining the version / path to repository folder.
Changing the config file by hand as you did above (and not running pantagruel --refresh init
afterwards) should solve the issue of version check and et you get on using the new version.
about the error with Prokka: I am a bit confused about it not liking the names it gave itself to the proteins... this might be due to inconsistencies between the version of Prokka you run and the BioPerl libraries that are installed on your server.
can you give a look at what the command readlink -f $(which prokka)
gives you please?
to avoid conflicts in perl libs, i would suggest installing prokka locally e.g. using homebrew
(which should pick up the right BioPerl libs) and using that local install.
if this is an issue to do that on your server, I would strongly suggest doing it on another machine like your laptop and then giving the proka annotation as input to pantagruel
As explained [here}(https://github.com/flass/pantagruel#input-data), you just need to put each genome annotation in its respective folder, themselves in the folder user_genomes/annotation/
.
Hi Florent,
Here is the version of Prokka: /apps/prokka/1.14/bin/prokka
There is one BIG problem with your solution. If I update the config file manually, Pantagruel won't run anymore. See error message below.
[wilkins@gorilla Panta]$ pantagruel -i /scratch/clamchatka/Panta/test8/environ_pantagruel_test8.sh fetch
This is Pantagruel pipeline version 9531df2e57fd032f1ff8e11b79091953833f978e using source code from repository '/apps/pantagruel/9531df2e57fd032f1ff8e11b79091953833f978e'
# will run tasks: 0
[2019-11-15 19:33:26] Pantagrel pipeline task 0: fetch public genome data from NCBI sequence databases and annotate private genomes.
ERROR: the current version of pantagruel (commit 9531df2) is different from the one used to generate the config file '/scratch/clamchatka/Panta/test9/environ_pantagruel_test9.sh' (commit 4867c04).
Please regenerate the config file with `pantagruel init` to ensure compatibility; for the same parameters to be set, just run the same command with same options as previously.
ERROR: Pantagrel pipeline task 0: failed.
And here is the config file itself:
#!/bin/bash
## Pantagruel database 'test9'
## built with Pantagruel version '9531df2e57fd032f1ff8e11b79091953833f978e'; source code available at 'https://github.com/flass/pantagruel'
# the init command with which the config file was created
ptginitcmd='pantagruel -d test9 -r /scratch/clamchatka/Panta/ -a /scratch/clamchatka/Panta/user_genomes/ -T /scratch/clamchatka/Panta/NCBI/Taxonomy_2019-11-12/ init'
# location (folder) of Pantagruel software that was used
export ptgrepo='/scratch/clamchatka/Panta/pantagruel'
# derive paths to Pantagruel scripts and Python modules
export ptgscripts='/scratch/clamchatka/Panta/pantagruel/scripts'
export PYTHONPATH='/scratch/clamchatka/Panta/pantagruel/python_libs'
# database parameters (primary variables)
export ptgroot='/scratch/clamchatka/Panta' # root folder where to build the database
export ptgdbname='test9' # name of dataabse
export ptgversinit='4867c048788ba7ec92dfd5ae9148d0349411151c' # current version of Pantagruel software
export myemail='undisclosed' # user identity (better use e-amil address)
export famprefix='PANTAG' # gene family prefix
export ncbitax='/scratch/clamchatka/Panta/NCBI/Taxonomy_2019-11-12' # folder of up-to-date NCBI Taxonomy database
export ncbiass='' # folder of RefSeq genomes to include in the study
export listncbiass='' # list of accessions of RefSeq genomes to include in the study
export customassemb='/scratch/clamchatka/Panta/user_genomes' # folder of custom genome assemblies to include in the study
export refass='' # folder of reference (RefSeq) genomes only to use as reference for the annotation of custom genome assemblies
export listrefass='' # list of accessions of reference (RefSeq) genomes only to use as reference for the annotation of custom genome assemblies
export coreseqtype='cds' # either 'cds' or 'protein'
export pseudocoremingenomes= # the minimum number of genomes in which a gene family should be present to be included in the pseudo-core genome gene set
export userreftree='' # possible user-provided reference tree
export poplgthresh='default' # parameter to define populations of genomes in the reference tree (stem branch length threshold, default value depends on coreseqtype)
export poplgleafmul='1.5' # parameter to define populations of genomes in the reference tree (multiplier to the former in case it is a leaf, default 1.5)
export popbsthresh='80' # parameter to define populations of genomes in the reference tree (stem branch support threshold, default 80)
export rootingmethod='treebalance'
export chaintype='fullgenetree' # whether gene trees will be collapsed ('collapsed', if -c option enabled) or not ('fullgenetree', default)
export genefamlist='' # list of gene families for which computation of gene trees and all subsequent analyses will be restricted (default: no restriction)
# non-default parameters for gene trees collapsing derived from -C option value (passed to init script via ${collapseCladeParams}):
export cladesupp=70 # - clade criterion trheshold (int)
export subcladesupp=35 # - wihtin-clade criterion trheshold (int)
export criterion='bs' # - criterion (branch support: 'bs', branch length 'lg')
export withinfun='median' # - aggregate function for testing within the clade ('min', 'max', 'mean', 'median')
export hpcremoteptgroot='none' # if not empty nor 'none', will use this server address to send data and scripts to run heavy computions there
# other parameters have default values defined in the generic source file environ_pantagruel_defaults.sh
source ${ptgscripts}/pipeline/environ_pantagruel_defaults.sh
# these defalts can be overriden by uncommenting the relevant line below and editing the variable's value
# default values are:
# Prokka annotation parameters (only relevant if custom genome assemblies are provided):
#~ export assembler="somesoftware"
#~ export seqcentre="somewhere"
#~ export refgenus="Reference"
# species tree inference parameters
#~ export ncorebootstrap=200
# gene tree inference parameters
#~ export mainresulttag='rootedTree'
# gene trees collapsing DEFAULT values (used when -C option is NOT present in init call)
#~ export cladesuppdef=70
#~ export subcladesuppdef=35
#~ export criteriondef='bs'
#~ export withinfundef='median'
# gene tree/species tree reconciliation inference parameters
#~ export ALEalgo='ALEml'
#~ export recsamplesize=1000
# gene tree/species tree reconciliation parsing parameters for co-evolution analysis
#~ export evtypeparse='ST'
#~ export minevfreqparse=0.1
#~ export minevfreqmatch=0.5
#~ export minjoinevfreqmatch=1.0
#~ export maxreftreeheight=0.25
# secondary vars are defined based on the above
source ${ptgscripts}/pipeline/environ_pantagruel_secondaryvars.sh
# load shared functions
source ${ptgscripts}/pipeline/pantagruel_pipeline_functions.sh
Hi Laetitia,
sorry for this long ping-pong of error tracking. I think this error of 'wrong version' is due to the fact you used the system-wide installed version by calling just pantagruel
instead of /scratch/clamchatka/Panta/pantagruel/pantagruel
(if I am correct that is where you cloned the last versions of the repository).
Please try again with the full path to the executable.
It is fixed and running.
Thank you so much for your patience, Florent. I hope others can benefit from our 20 ball long ping pong rally. https://www.youtube.com/watch?v=jlBOsGASWPI
Hi Florent, this is a more specific issue.
I have manually downloaded the Taxonomy files as described in issue #24 for for Matt Bawn (https://github.com/flass/pantagruel/issues/24#issuecomment-550270892). I also had to manually install your module "tree2".
Then, I ran the following commands:
Is this a problem with permissions? The file it claims missing is there!
See here:
I cannot trace back what is going wrong here.