steineggerlab / ufcg

UFCG: Universal Fungal Core Genes
https://ufcg.steineggerlab.com
GNU General Public License v3.0
29 stars 0 forks source link

Augustus takes 420G ram +59h #5

Open Sebastien-Raguideau opened 1 year ago

Sebastien-Raguideau commented 1 year ago

Hello I have a new issue for you :) Some of the genomes I'm looking at are taking forever to be profiled. When looking at it in detail augustus is being slow (50h+) as well as hogging ram (worst so far is 420G ram) on some genes. As I don't monitor all the time I don't know if some of theses managed to complete before. From what Is running in my server I have 6 ELP3 and 3 RPO21 running for more than 40h.

I can share with you an example where it does not end so far. Here is a command running forever: augustus --optCfgFile=/mnt/gpfs/seb/Applications/UFCG_v1.0.1/UFCG/config/ppx.cfg --predictionStart=165215 --predictionEnd=185215 --proteinprofile=/mnt/gpfs/seb/Applications/UFCG_v1.0.1/UFCG/config/model/pro/ELP3.hmm /home/sebr/seb/Database/Fungus_DB/UFCG/tmp/Conco1/tmp/Conco1/Conco1_scaffold_37.fna

with genome and temp dir: Conco1_issue.tar.gz

endixk commented 1 year ago

Hello again,

This is a weird behavior of augustus that I also witnessed before. I eventually failed to find the main cause or to determine the situation in which such behavior is observed.

As this issue will persist, from the next release, a timeout will be wrapped around the augustus predictions to prevent the pipeline from consuming unnecessary time or resources.

To root out the primary cause, the issue could be redirected to the Augustus repository, where the developers of augustus can actually deal with this unexpected behavior.

endixk commented 1 year ago

Now the same problem won't happen with the newest version, but will handle the 5-minute timeouts with messages something like this:

WARN  |:  Command timed out: augustus --optCfgFile=... --predictionStart=14960 --predictionEnd=34960
--proteinprofile=... /tmp/Conco1/Conco1_scaffold_199.fna > /tmp/Conco1/Conco1_scaffold_199_p24960_RPO21.gff