Closed kushalsuryamohan closed 1 year ago
hyphaltip
commented
1 year ago how much memory did you allocate to the job - the error message indicates a memory exceeding message.
This may be an issue with exonerate polishing very long alignments - can you see what is in the logfiles folder - there is a funannotate-p2g.log and a funannotate-predict.log file might give some clues as to where it is stuck.
nextgenusfs
commented
1 year ago I think that python when running multiprocessing with os.fork() will actually duplicate all items in RAM per CPU. So you might actually set to lower number of CPUs and see if it gets through. I guess this isn't "LowMemory" enough for the larger genomes....it is dying actually before it even gets to the protein2genome stage, this function is just trying to calculate and write a bed file of soft masked regions:
ContigSizes, GenomeLength, maskedSize, percentMask = lib.checkMasklowMem()
kushalsuryamohan
commented
1 year ago Hi @hyphaltip, here is my sbatch script:
#!/bin/sh
#SBATCH -J funannotate
#SBATCH -n 50
#SBATCH -e funannotate_predict.err
#SBATCH -o funannotate_predict.out
#SBATCH --mem-per-cpu=4G
source /MG/SHARED/APPS/ANACONDA_DIR/anaconda/bin/activate funannotate
export FUNANNOTATE_DB=/MG/SHARED/APPS/FUNANNOTATE_DB
export GENEMARK_PATH=/MG/SHARED/APPS/GENEMARK_LISCENCE/gmes_linux_64_4/
export PATH=/MG/SHARED/APPS/GENEMARK_LISCENCE/gmes_linux_64_4/:$PATH
funannotate predict -i Boomslang_final_cleaned_renamed.fasta \
-o funannotate_train -s "Boomslang_final_cleaned_renamed" --no-progress --cpus 45And here are the contents of the funannotate-p2g.log:
[05/11/23 11:29:15]: /MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/lib/python3.8/site-packages/funannotate/aux_scripts/funannotate-p2g.py -p funannotate_train/predict_misc/proteins.combined.fa -g /MG/SHARED/ANALYSIS/DEMUX/GENOME_ANALYSIS/Boomslang/funannotate_train/predict_misc/genome.softmasked.fa -o funannotate_train/predict_misc/protein_alignments.gff3 --maxintron 3000 --cpus 16 --exonerate_pident 80 --ploidy 1 -f diamond --tmpdir /tmp --tblastn_out funannotate_train/predict_misc/p2g.diamond.out --logfile funannotate_train/logfiles/funannotate-p2g.log
[05/11/23 11:29:17]: Mapping 556,668 proteins to genome using diamond and exonerate [05/11/23 11:29:17]: Diamond v2.1.6; Exonerate v2.4.0 [05/11/23 11:29:17]: diamond makedb --threads 16 --in /MG/SHARED/ANALYSIS/DEMUX/GENOME_ANALYSIS/Boomslang/funannotate_train/predict_misc/proteins.combined.fa --db diamond [05/11/23 11:29:21]: diamond blastx --threads 16 -q /MG/SHARED/ANALYSIS/DEMUX/GENOME_ANALYSIS/Boomslang/funannotate_train/predict_misc/genome.softmasked.fa --db diamond -o diamond.matches.tab -e 1e-10 -k 0 --more-sensitive --unal 0 -c 1 -F 15 -f 6 sseqid slen sstart send qseqid qlen qstart qend pident length evalue score qcovhsp qframe [05/11/23 11:59:10]: Writing 184478 contig splits for exonerate [05/11/23 11:59:32]: Found 184,478 preliminary alignments with diamond in 0:27:34 --> generated FASTA files for exonerate in 0:02:40
And lastly, the contents of funannotate-predict.log:
06/02/23 12:34:29: /MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/bin/funannotate predict -i Boomslang_final_cleaned_renamed.fasta -o funannotate_train -s Boomslang_final_cleaned_renamed --no-progress --cpus 45
06/02/23 12:34:29: OS: CentOS Linux 7, 104 cores, ~ 791 GB RAM. Python: 3.8.15 06/02/23 12:34:29: Running funannotate v1.8.15 06/02/23 12:34:29: GeneMark path: /MG/SHARED/APPS/GENEMARK_LISCENCE/gmes_linux_64_4/ 06/02/23 12:34:29: Full path to gmes_petap.pl: /MG/SHARED/APPS/GENEMARK_LISCENCE/gmes_linux_64_4/gmes_petap.pl 06/02/23 12:34:29: GeneMark appears to be functional? True 06/02/23 12:34:29: exonerate version=exonerate 2.4.0 path=/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/bin/exonerate 06/02/23 12:34:29: diamond version=2.1.6 path=/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/bin/diamond 06/02/23 12:34:29: tbl2asn version=25.8 path=/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/bin/tbl2asn 06/02/23 12:34:29: bedtools version=bedtools v2.30.0 path=/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/bin/bedtools 06/02/23 12:34:29: augustus version=3.5.0 path=/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/bin/augustus 06/02/23 12:34:29: etraining version=NA path=/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/bin/etraining 06/02/23 12:34:29: tRNAscan-SE version=2.0.11 (Oct 2022) path=/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/bin/tRNAscan-SE 06/02/23 12:34:29: bam2hints version=NA path=/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/bin/bam2hints 06/02/23 12:34:29: minimap2 version=2.26-r1175 path=/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/bin/minimap2
06/02/23 12:34:29: Found training files, will re-use these files: --rna_bam funannotate_train/training/funannotate_train.coordSorted.bam --pasa_gff funannotate_train/training/funannotate_train.pasa.gff3 --stringtie funannotate_train/training/funannotate_train.stringtie.gtf --transcript_alignments funannotate_train/training/funannotate_train.transcripts.gff3 06/02/23 12:34:29: {'augustus': 1, 'hiq': 2, 'genemark': 1, 'pasa': 6, 'codingquarry': 2, 'snap': 1, 'glimmerhmm': 1, 'proteins': 1, 'transcripts': 1} [06/02/23 12:34:32]: {'augustus': 'pasa', 'genemark': 'selftraining', 'snap': 'pasa', 'glimmerhmm': 'pasa', 'codingquarry': 'rna-bam'} [06/02/23 12:34:32]: Parsed training data, run ab-initio gene predictors as follows: [06/02/23 12:34:32]: augustus --species=anidulans --proteinprofile=/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/lib/python3.8/site-packages/funannotate/config/EOG092C0B3U.prfl /MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/lib/python3.8/site-packages/funannotate/config/busco_test.fa [06/02/23 12:34:35]: perl /MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/opt/evidencemodeler-1.1.1/EvmUtils/gff3_gene_prediction_file_validator.pl /MG/SHARED/ANALYSIS/DEMUX/GENOME_ANALYSIS/Boomslang/funannotate_train/predict_misc/pasa_predictions.gff3 [06/02/23 12:34:38]: {'augustus': 1, 'hiq': 2, 'genemark': 1, 'pasa': 6, 'codingquarry': 2, 'snap': 1, 'glimmerhmm': 1, 'proteins': 1, 'transcripts': 1} [06/02/23 12:43:49]: Loading genome assembly and parsing soft-masked repetitive sequences
Is there a recommended fix to this issue I'm encountering? Appreciate your help!
Thanks!
kushalsuryamohan
commented
1 year ago I tried to run
funannotate predictusing 12 cpus on a SLURM HPC cluster, despite running for ~21 days, the program was stuck at the alignment step. I assumed this might be a memory-related error so I canceled the job and resumed using 48 cpus. However, now I get a memory allocation error.
Hi @nextgenusfs, as I mentioned in my original post, I tried to run funannotate predict using 12 cpus on a SLURM HPC cluster. This left me in a situation where the job was running for ~21 days and the program was stuck at the alignment step (xxx jobs finished, y failed, xxxxxxxx left) and not progressing. I assumed this might be a memory-related error so I canceled the job and resumed using 48 cpus. However, now I see this memory allocation error. I've now resumed the job with 24 cpus and I'm opting not to write long sub job logs.
kushalsuryamohan
commented
1 year ago Update: The job with 24 cpus seems to have progressed beyond what I was seeing previously when using 48 cpus. See below. Based on the memory allocated (and ~60k transcripts and the ~550k proteins to be mapped to the genome), is there a timeframe that I can expect the predict step to complete?
I'll keep you posted in case I encounter another error.
[Jun 02 01:56 PM]: OS: CentOS Linux 7, 104 cores, ~ 791 GB RAM. Python: 3.8.15 [Jun 02 01:56 PM]: Running funannotate v1.8.15 [Jun 02 01:56 PM]: Found training files, will re-use these files: --rna_bam funannotate_train/training/funannotate_train.coordSorted.bam --pasa_gff funannotate_train/training/funannotate_train.pasa.gff3 --stringtie funannotate_train/training/funannotate_train.stringtie.gtf --transcript_alignments funannotate_train/training/funannotate_train.transcripts.gff3 [Jun 02 01:56 PM]: Parsed training data, run ab-initio gene predictors as follows: Program Training-Method augustus pasa
codingquarry rna-bam
genemark selftraining
glimmerhmm pasa
snap pasa
[Jun 02 02:05 PM]: Loading genome assembly and parsing soft-masked repetitive sequences [Jun 02 02:08 PM]: Genome loaded: 3,111 scaffolds; 1,767,527,460 bp; 53.75% repeats masked [Jun 02 02:08 PM]: Parsed 58,667 transcript alignments from: funannotate_train/training/funannotate_train.transcripts.gff3 [Jun 02 02:08 PM]: Creating transcript EVM alignments and Augustus transcripts hintsfile [Jun 02 02:08 PM]: Existing RNA-seq BAM hints found: funannotate_train/predict_misc/hints.BAM.gff [Jun 02 02:08 PM]: Mapping 556,668 proteins to genome using diamond and exonerate
kushalsuryamohan
commented
1 year ago Hello @hyphaltip and @nextgenusfs, GeneMark-ES has completed for my genome of interest. However, I encounter this error now and the program exited. I see this issue was faced by another user but that remains open on Github. Any idea why this may be? I checked the file permissions and they seem okay.
[Jun 02 01:56 PM]: OS: CentOS Linux 7, 104 cores, ~ 791 GB RAM. Python: 3.8.15 [Jun 02 01:56 PM]: Running funannotate v1.8.15 [Jun 02 01:56 PM]: Found training files, will re-use these files: --rna_bam funannotate_train/training/funannotate_train.coordSorted.bam --pasa_gff funannotate_train/training/funannotate_train.pasa.gff3 --stringtie funannotate_train/training/funannotate_train.stringtie.gtf --transcript_alignments funannotate_train/training/funannotate_train.transcripts.gff3 [Jun 02 01:56 PM]: Parsed training data, run ab-initio gene predictors as follows: ESC[4mProgram Training-MethodESC[0m augustus pasa
codingquarry rna-bam
genemark selftraining
glimmerhmm pasa
snap pasa
[Jun 02 02:05 PM]: Loading genome assembly and parsing soft-masked repetitive sequences [Jun 02 02:08 PM]: Genome loaded: 3,111 scaffolds; 1,767,527,460 bp; 53.75% repeats masked [Jun 02 02:08 PM]: Parsed 58,667 transcript alignments from: funannotate_train/training/funannotate_train.transcripts.gff3 [Jun 02 02:08 PM]: Creating transcript EVM alignments and Augustus transcripts hintsfile [Jun 02 02:08 PM]: Existing RNA-seq BAM hints found: funannotate_train/predict_misc/hints.BAM.gff [Jun 02 02:08 PM]: Mapping 556,668 proteins to genome using diamond and exonerate [Jun 02 02:34 PM]: Found 184,478 preliminary alignments with diamond in 0:23:53 --> generated FASTA files for exonerate in 0:02:15 [Jun 02 02:47 PM]: Exonerate finished in 0:12:59: found 4,373 alignments [Jun 02 02:48 PM]: Running GeneMark-ES on assembly [Jun 02 07:38 PM]: 64,800 predictions from GeneMark [Jun 02 07:38 PM]: Filtering PASA data for suitable training set Traceback (most recent call last): File "/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/bin/funannotate", line 10, insys.exit(main()) File "/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/lib/python3.8/site-packages/funannotate/funannotate.py", line 716, in main mod.main(arguments) File "/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/lib/python3.8/site-packages/funannotate/predict.py", line 2020, in main lib.runSubprocess( File "/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/lib/python3.8/site-packages/funannotate/library.py", line 792, in runSubprocess process = subprocess.run( File "/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/lib/python3.8/subprocess.py", line 493, in run with Popen(*popenargs, **kwargs) as process: File "/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/lib/python3.8/subprocess.py", line 858, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/lib/python3.8/subprocess.py", line 1704, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) PermissionError: [Errno 13] Permission denied: 'filterGenemark.pl'
And here are the file permissions:
-bash-4.2$ ls -lrt /MG/SHARED/APPS/ANACONDA_DIR/anaconda/envs/funannotate/lib/python3.8/site-packages/funannotate/aux_scripts total 624 -rwxrwxr-x 2 nitin.m ngsmdgrp 723 Apr 26 08:43 xmlcombine.py -rwxrwxr-x 2 nitin.m ngsmdgrp 5461 Apr 26 08:43 trnascan2gff3.pl -rwxrwxr-x 2 nitin.m ngsmdgrp 8821 Apr 26 08:43 trinity.py -rwxrwxr-x 2 nitin.m ngsmdgrp 9600 Apr 26 08:43 tbl2asn_parallel.py -rwxrwxr-x 2 nitin.m ngsmdgrp 389 Apr 26 08:43 sam2bam.sh -rwxrwxr-x 2 nitin.m ngsmdgrp 15288 Apr 26 08:43 runIPRscan.py -rwxrwxr-x 2 nitin.m ngsmdgrp 35348 Apr 26 08:43 phobius-remote.pl -rwxrwxr-x 2 nitin.m ngsmdgrp 3877 Apr 26 08:43 phobius-multiproc.py -rwxrwxr-x 2 nitin.m ngsmdgrp 65930 Apr 26 08:43 pal2nal.pl -rwxrwxr-x 2 nitin.m ngsmdgrp 13686 Apr 26 08:43 iprscan-local.py -rwxrwxr-x 2 nitin.m ngsmdgrp 4600 Apr 26 08:43 iprscan2annotations.py -rwxrwxr-x 2 nitin.m ngsmdgrp 8357 Apr 26 08:43 hmmer_parallel.py -rwxrwxr-x 2 nitin.m ngsmdgrp 642 Apr 26 08:43 getEggNog.sh -rwxrwxr-x 2 nitin.m ngsmdgrp 4658 Apr 26 08:43 genemark_gtf2gff3.pl -rwxrwxr-x 2 nitin.m ngsmdgrp 23323 Apr 26 08:43 funannotate-runEVM.py -rwxrwxr-x 2 nitin.m ngsmdgrp 17422 Apr 26 08:43 funannotate-p2g.py -rwxrwxr-x 2 nitin.m ngsmdgrp 144316 Apr 26 08:43 funannotate-BUSCO2-py2.py -rwxrwxr-x 2 nitin.m ngsmdgrp 144435 Apr 26 08:43 funannotate-BUSCO2.py -rwxrwxr-x 2 nitin.m ngsmdgrp 5716 Apr 26 08:43 filterIntronsFindStrand.pl -rwxrwxr-x 2 nitin.m ngsmdgrp 21217 Apr 26 08:43 filterGenemark.pl -rwxrwxr-x 2 nitin.m ngsmdgrp 5716 Apr 26 08:43 fasta2agp.py -rwxrwxr-x 2 nitin.m ngsmdgrp 2675 Apr 26 08:43 enrichment_parallel.py -rwxrwxr-x 2 nitin.m ngsmdgrp 6941 Apr 26 08:43 augustus_parallel.py drwxr-xr-x 2 nitin.m ngsmdgrp 4096 May 8 22:47 pycache -bash-4.2$
The sysadmin installed funannotate and dependencies but since all users have read and execute permissions, I can't seem to identify the issue here. Any help is appreciated.
Thanks!
kushalsuryamohan
commented
1 year ago I was able to fix this within predict.py. Closing for now. Thanks!
Hello Jon @nextgenusfs, I'm trying to run funannotate predict for a reptilian genome (~1.5-2 Gb). This genome was de novo assembled using long read sequencing (PacBio HiFi reads) and Hi-C to yield a fairly good assembly (N50 ~250 Mb). Following clean up and renaming of fasta headers, I was able to run
funannotate trainusing RNA-seq data. The resulting PASA and trinity outputs were then used for funannotate predict.When I tried to run
funannotate predictusing 12 cpus on a SLURM HPC cluster, despite running for ~21 days, the program was stuck at the alignment step. I assumed this might be a memory-related error so I canceled the job and resumed using 48 cpus. However, now I get a memory allocation error.Can you please help?
Appreciate your help.
Thanks!
Are you using the latest release? Yes. v1.8.15
Describe the bug Error when running funannotate predict
What command did you issue?
funannotate predict -i Boomslang_final_cleaned_renamed.fasta -o funannotate_train -s "Boomslang_final_cleaned_renamed" --no-progress --cpus 48Logfiles
OS/Install Information
funannotate check --show-versions(funannotate) -bash-4.2$ funannotate check --show-versions