myco-bradshaw
commented
2 years ago Closed AlWa1 closed 2 years ago
myco-bradshaw
commented
2 years ago 
AlWa1
commented
2 years ago 724
Thanks for the link - I have been trying to fix the bug according to this here but this did not unfortunately work for me. Any other suggestions?
hyphaltip
commented
2 years ago I just updated the glimmerhmm package on bioconda - https://github.com/bioconda/bioconda-recipes/pull/37372 - the issue is the build 2 of this package from 2019 does not update the shebang line or the use lib line which was needed - build 3 did but it seems to not be distributing from bioconda due to some failed tests and i think problems with ftp -> http changes at the source distribution. I've provided a patch and pull request which I think solves this so as soon as this makes it into the upstream bioconda ecosystem i think it will be possible to get this version installed and it will fix the orf.pm missing problem.
make sure you are installed a version of the package that is build > 2 (eg the last number in the info is the build number- in this case it is build 5 the one I just put up glimmerhmm:3.0.4--pl5321h87f3376_5
hyphaltip
commented
2 years ago can you confirm if this will work for you now that the glimmerhmm that is available in bioconda should have the perl library loading fixes in the recipe.
myco-bradshaw
commented
2 years ago can you confirm if this will work for you now that the glimmerhmm that is available in bioconda should have the perl library loading fixes in the recipe.
I cannot speak for OP, but performing a mamba update glimmerhmm fixed this issue for my local conda based install.
AlWa1
commented
2 years ago Thanks, the glimmerhmm update helped to progress in the test run, but now I get the same issue as mentioned in here #796 :
Could not locate command line parameters file: annotate/predict_misc/ab_initio_parameters/augustus/parameters/aug_cmdln_parameters.json.
I have the feeling that this might be an issue for some people with Augustus 3.5.0?
Here the full log file:
You are running Python v 3.8.13. Now checking python packages... biopython: 1.79 goatools: 1.2.3 matplotlib: 3.4.3 natsort: 8.2.0 numpy: 1.23.4 pandas: 1.5.1 psutil: 5.9.3 requests: 2.28.1 scikit-learn: 1.1.2 scipy: 1.9.3 seaborn: 0.12.1 All 11 python packages installed
You are running Perl v b'5.032001'. Now checking perl modules... Carp: 1.50 Clone: 0.42 DBD::SQLite: 1.70 DBD::mysql: 4.046 DBI: 1.643 DB_File: 1.855 Data::Dumper: 2.183 File::Basename: 2.85 File::Which: 1.24 Getopt::Long: 2.52 Hash::Merge: 0.302 JSON: 4.10 LWP::UserAgent: 6.67 Logger::Simple: 2.0 POSIX: 1.94 Parallel::ForkManager: 2.02 Pod::Usage: 1.69 Scalar::Util::Numeric: 0.40 Storable: 3.15 Text::Soundex: 3.05 Thread::Queue: 3.13 Tie::File: 1.06 URI::Escape: 5.12 YAML: 1.30 local::lib: 2.000029 threads: 2.21 threads::shared: 1.61 All 27 Perl modules installed
[Oct 24 12:14 PM]: OS: Scientific Linux 7.7, 32 cores, ~ 395 GB RAM. Python: 3.8.13
[Oct 24 12:14 PM]: Running funannotate v1.8.13
[Oct 24 12:14 PM]: Skipping CodingQuarry as no --rna_bam passed
[Oct 24 12:14 PM]: Parsed training data, run ab-initio gene predictors as follows:
[4mProgram Training-Method[0m
augustus pretrained
genemark selftraining
glimmerhmm busco
snap busco
[Oct 24 12:14 PM]: Loading genome assembly and parsing soft-masked repetitive sequences
[Oct 24 12:14 PM]: Genome loaded: 6 scaffolds; 3,776,588 bp; 19.75% repeats masked
[Oct 24 12:14 PM]: Mapping 1,065 proteins to genome using diamond and exonerate
[Oct 24 12:14 PM]: Found 1,505 preliminary alignments with diamond in 0:00:01 --> generated FASTA files for exonerate in 0:00:00
Progress: 0.00%
Progress: 3.65%
Progress: 13.49%
Progress: 27.11%
Progress: 46.31%
Progress: 58.47%
Progress: 70.03%
Progress: 82.26%
Progress: 90.83%
Progress: 99.93%
Progress: 99.93%
Progress: 99.93%
Progress: 99.93%
Progress: 99.93%
Progress: 99.93%
[Oct 24 12:15 PM]: Exonerate finished in 0:00:15: found 1,270 alignments
[Oct 24 12:15 PM]: Running GeneMark-ES on assembly
[Oct 24 12:17 PM]: 1,560 predictions from GeneMark
[Oct 24 12:17 PM]: Running BUSCO to find conserved gene models for training ab-initio predictors
[Oct 24 12:22 PM]: 370 valid BUSCO predictions found, validating protein sequences
[Oct 24 12:22 PM]: 367 BUSCO predictions validated
[Oct 24 12:22 PM]: Running Augustus gene prediction using saccharomyces parameters
Progress: 0.00%
[Oct 24 12:22 PM]: CMD ERROR: augustus --species=saccharomyces --predictionStart=346841.7857267572 --predictionEnd=703683.5714535144 --extrinsicCfgFile=/rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/lib/python3.8/site-packages/funannotate/config/extrinsic.E.XNT.RM.cfg --hintsfile=annotate/predict_misc/hints.ALL.gff --AUGUSTUS_CONFIG_PATH=annotate/predict_misc/ab_initio_parameters/augustus --softmasking=1 --gff3=on --UTR=off --stopCodonExcludedFromCDS=False augustus_tmp_f638cb64-6dd6-4a19-97ed-40e9f2b62553/CP022972.1.fa
[Oct 24 12:22 PM]:
augustus: ERROR
Could not locate command line parameters file: annotate/predict_misc/ab_initio_parameters/augustus/parameters/aug_cmdln_parameters.json.
nextgenusfs
commented
2 years ago Please try to update funannotate to the latest in master with pip, the augustus >=3.4 is non-functional with the latest release (I think).
AlWa1
commented
2 years ago Brilliant, problem solved for me - thanks @nextgenusfs and @hyphaltip for your advice!
Are you using the latest release? Yes, installed fun annotate via conda (v1.7.4, installed 4 days ago)
Describe the bug
After having installed fun annotate via conda I have been trying to run the funannotate test command
funannotate test -t all --cpus 8
which lead to following mistake (see full log file below):
CMD ERROR: trainGlimmerHMM /rds/project/ss2123/rds-ss2123-team_seb_storage/projects/20220915_fun_test/test-predict_1a5e2c98-879a-4df9-953a-7ccdba2a7d30/annotate/predict_misc/genome.softmasked.fa /rds/project/ss2123/rds-ss2123-team_seb_storage/projects/20220915_fun_test/test-predict_1a5e2c98-879a-4df9-953a-7ccdba2a7d30/annotate/predict_misc/glimmer.exons -d annotate/predict_misc/glimmerhmm
Can't locate orf.pm in @INC (@INC contains: /rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/bin /home/aw932/perl5/lib/perl5/5.16.3/x86_64-linux-thread-multi /home/aw932/perl5/lib/perl5/5.16.3 /home/aw932/perl5/lib/perl5/x86_64-linux-thread-multi /home/aw932/perl5/lib/perl5 /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/bin/trainGlimmerHMM line 16.
BEGIN failed--compilation aborted at /rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/bin/trainGlimmerHMM line 16.
What command did you issue? Copy/paste the command used.
Logfiles
######################################################### Running
funannotate cleanunit testing: minimap2 mediated assembly duplications Downloading: https://osf.io/8pjbe/download?version=1 Bytes: 252076 8192 [100.00%]6 input contigs, 6 larger than 500 bp, N50 is 427,039 bp Checking duplication of 6 contigs
minimap2 version=2.24-r1122 path=/rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/bin/minimap2 scaffold_73 appears duplicated: 100% identity over 100% of the contig. contig length: 15153 scaffold_91 appears duplicated: 100% identity over 100% of the contig. contig length: 8858 scaffold_27 appears duplicated: 100% identity over 100% of the contig. contig length: 427039
6 input contigs; 6 larger than 500 bp; 3 duplicated; 3 written to file CMD: funannotate clean -i test.clean.fa -o test.exhaustive.fa --exhaustive ######################################################### ######################################################### SUCCESS:
funannotate cleantest complete. ################################################################################################################## Running
funannotate maskunit testing: RepeatModeler --> RepeatMasker Downloading: https://osf.io/hbryz/download?version=1 Bytes: 375687 8192 [100.00%][Sep 22 07:47 PM]: OS: Scientific Linux 7.7, 32 cores, ~ 395 GB RAM. Python: 3.8.13 [Sep 22 07:47 PM]: Running funanotate v1.8.13 [Sep 22 07:47 PM]: Soft-masking simple repeats with tantan [Sep 22 07:47 PM]: Repeat soft-masking finished: Masked genome: /rds/project/ss2123/rds-ss2123-team_seb_storage/projects/20220915_fun_test/test-mask_1a5e2c98-879a-4df9-953a-7ccdba2a7d30/test.masked.fa num scaffolds: 2 assembly size: 1,216,048 bp masked repeats: 50,965 bp (4.19%)CMD: funannotate mask -i test.fa -o test.masked.fa --cpus 8 ######################################################### ######################################################### SUCCESS:
funannotate masktest complete. ################################################################################################################## Running
funannotate predictunit testing Downloading: https://osf.io/te2pf/download?version=1 Bytes: 1489808 8192[Sep 22 07:47 PM]: OS: Scientific Linux 7.7, 32 cores, ~ 395 GB RAM. Python: 3.8.13 [Sep 22 07:47 PM]: Running funannotate v1.8.13 [Sep 22 07:47 PM]: Skipping CodingQuarry as no --rna_bam passed [Sep 22 07:47 PM]: Parsed training data, run ab-initio gene predictors as follows: [4mProgram Training-Method[0m augustus pretrained
genemark selftraining
glimmerhmm busco
snap busco
[Sep 22 07:47 PM]: Loading genome assembly and parsing soft-masked repetitive sequences [Sep 22 07:47 PM]: Genome loaded: 6 scaffolds; 3,776,588 bp; 19.75% repeats masked [Sep 22 07:47 PM]: Mapping 1,065 proteins to genome using diamond and exonerate [Sep 22 07:47 PM]: Found 1,505 preliminary alignments with diamond in 0:00:01 --> generated FASTA files for exonerate in 0:00:00 Progress: 0.00% Progress: 3.26% Progress: 11.30% Progress: 21.26% Progress: 35.75% Progress: 50.37% Progress: 61.66% Progress: 70.56% Progress: 81.20% Progress: 88.44% Progress: 99.93% Progress: 99.93% Progress: 99.93% Progress: 99.93% Progress: 99.93% Progress: 99.93% Progress: 99.93% Progress: 99.93% Progress: 99.93% Progress: 99.93% Progress: 99.93% Progress: 99.93% Progress: 99.93% [Sep 22 07:48 PM]: Exonerate finished in 0:00:23: found 1,270 alignments [Sep 22 07:48 PM]: Running GeneMark-ES on assembly [Sep 22 07:49 PM]: 1,564 predictions from GeneMark [Sep 22 07:49 PM]: Running BUSCO to find conserved gene models for training ab-initio predictors [Sep 22 07:56 PM]: 370 valid BUSCO predictions found, validating protein sequences [Sep 22 07:56 PM]: 367 BUSCO predictions validated [Sep 22 07:56 PM]: Running Augustus gene prediction using saccharomyces parameters Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 0.00% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 27.27% Progress: 36.36% Progress: 36.36% Progress: 45.45% Progress: 45.45% Progress: 45.45% Progress: 45.45% Progress: 45.45% Progress: 54.55% Progress: 54.55% Progress: 63.64% Progress: 72.73% Progress: 81.82% Progress: 81.82% Progress: 81.82% Progress: 81.82% Progress: 81.82% Progress: 81.82% Progress: 81.82% Progress: 81.82% Progress: 81.82% Progress: 81.82% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% Progress: 90.91% [Sep 22 07:58 PM]: 1,485 predictions from Augustus [Sep 22 07:58 PM]: Pulling out high quality Augustus predictions [Sep 22 07:58 PM]: Found 371 high quality predictions from Augustus (>90% exon evidence) [Sep 22 07:58 PM]: Running SNAP gene prediction, using training data: annotate/predict_misc/busco.final.gff3 [Sep 22 07:59 PM]: 1,536 predictions from SNAP [Sep 22 07:59 PM]: Running GlimmerHMM gene prediction, using training data: annotate/predict_misc/busco.final.gff3 [Sep 22 07:59 PM]: CMD ERROR: trainGlimmerHMM /rds/project/ss2123/rds-ss2123-team_seb_storage/projects/20220915_fun_test/test-predict_1a5e2c98-879a-4df9-953a-7ccdba2a7d30/annotate/predict_misc/genome.softmasked.fa /rds/project/ss2123/rds-ss2123-team_seb_storage/projects/20220915_fun_test/test-predict_1a5e2c98-879a-4df9-953a-7ccdba2a7d30/annotate/predict_misc/glimmer.exons -d annotate/predict_misc/glimmerhmm [Sep 22 07:59 PM]: Can't locate orf.pm in @INC (@INC contains: /rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/bin /home/aw932/perl5/lib/perl5/5.16.3/x86_64-linux-thread-multi /home/aw932/perl5/lib/perl5/5.16.3 /home/aw932/perl5/lib/perl5/x86_64-linux-thread-multi /home/aw932/perl5/lib/perl5 /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/bin/trainGlimmerHMM line 16. BEGIN failed--compilation aborted at /rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/bin/trainGlimmerHMM line 16.
CMD: funannotate predict -i test.softmasked.fa --protein_evidence protein.evidence.fasta -o annotate --augustus_species saccharomyces --cpus 8 --species Awesome testicus ######################################################### ######################################################### Traceback (most recent call last): File "/rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/bin/funannotate", line 10, in
sys.exit(main())
File "/rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/lib/python3.8/site-packages/funannotate/funannotate.py", line 716, in main
mod.main(arguments)
File "/rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/lib/python3.8/site-packages/funannotate/test.py", line 405, in main
runPredictTest(args)
File "/rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/lib/python3.8/site-packages/funannotate/test.py", line 160, in runPredictTest
assert 1500 <= countGFFgenes(os.path.join(
File "/rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/lib/python3.8/site-packages/funannotate/test.py", line 45, in countGFFgenes
with open(input, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'test-predict_1a5e2c98-879a-4df9-953a-7ccdba2a7d30/annotate/predict_results/Awesome_testicus.gff3'
Thu 22 Sep 19:59:00 BST 2022
OS/Install Information
Checking dependencies for 1.8.13
You are running Python v 3.8.13. Now checking python packages... biopython: 1.79 goatools: 1.2.3 matplotlib: 3.4.3 natsort: 8.2.0 numpy: 1.23.3 pandas: 1.5.0 psutil: 5.9.2 requests: 2.28.1 scikit-learn: 1.1.2 scipy: 1.9.1 seaborn: 0.12.0 All 11 python packages installed
You are running Perl v b'5.032001'. Now checking perl modules... Carp: 1.50 Clone: 0.42 DBD::SQLite: 1.70 DBD::mysql: 4.046 DBI: 1.643 DB_File: 1.855 Data::Dumper: 2.183 File::Basename: 2.85 File::Which: 1.24 Getopt::Long: 2.52 Hash::Merge: 0.302 JSON: 4.09 LWP::UserAgent: 6.67 Logger::Simple: 2.0 POSIX: 1.94 Parallel::ForkManager: 2.02 Pod::Usage: 1.69 Scalar::Util::Numeric: 0.40 Storable: 3.15 Text::Soundex: 3.05 Thread::Queue: 3.13 Tie::File: 1.06 URI::Escape: 5.12 YAML: 1.30 local::lib: 2.000029 threads: 2.21 threads::shared: 1.61 All 27 Perl modules installed
Checking Environmental Variables... $FUNANNOTATE_DB=/rds/user/aw932/hpc-work/databases/funannotate_db $PASAHOME=/rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/opt/pasa-2.5.2 $TRINITY_HOME=/rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/opt/trinity-2.8.5 $EVM_HOME=/rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/opt/evidencemodeler-1.1.1 $AUGUSTUS_CONFIG_PATH=/rds/user/aw932/hpc-work/software/miniconda3_V2/envs/funannotate/config/ $GENEMARK_PATH=/home/aw932/rds/hpc-work/software/gmes All 6 environmental variables are set
Checking external dependencies... PASA: 2.5.2 CodingQuarry: 2.0 Trinity: 2.8.5 augustus: 3.4.0 bamtools: bamtools 2.5.1 bedtools: bedtools v2.30.0 blat: BLAT v35 diamond: 2.0.15 ete3: 3.1.2 exonerate: exonerate 2.4.0 fasta: no way to determine glimmerhmm: 3.0.4 gmap: 2021-08-25 gmes_petap.pl: 4.69_lic hisat2: 2.2.1 hmmscan: HMMER 3.3.2 (Nov 2020) hmmsearch: HMMER 3.3.2 (Nov 2020) java: 17.0.3-internal kallisto: 0.46.1 mafft: v7.508 (2022/Sep/07) makeblastdb: makeblastdb 2.2.31+ minimap2: 2.24-r1122 proteinortho: 6.1.1 pslCDnaFilter: no way to determine salmon: salmon 0.14.1 samtools: samtools 1.15.1 signalp: 4.1 snap: 2006-07-28 stringtie: 2.2.1 tRNAscan-SE: 2.0.9 (July 2021) tantan: tantan 39 tbl2asn: no way to determine, likely 25.X tblastn: tblastn 2.2.31+ trimal: trimAl v1.4.rev15 build[2013-12-17] trimmomatic: [0.003s][warning][os,container] Duplicate cpuset controllers detected. Picking /sys/fs/cgroup/cpuset, skipping /cgroup-sl/cpuset. ERROR: emapper.py not installed ERROR: pigz not installed