search

nathanweeks / exonerate

A fork of exonerate: a generic tool for sequence alignment
GNU General Public License v3.0
60 stars 25 forks source link

Killed automaticly #31

Open AndyLy2Zy opened 1 day ago

AndyLy2Zy commented 1 day ago

Hello All,

I've encountered a error while running the exonerate on HPC: _slurmstepd: error: Detected 12 oom-kill event(s) in StepId=16198179.batch. Some of your processes may have been killed by the cgroup out-of-memory handler. I tried many parameters but the errors still here.

Is there have some suggestions ? Thanks in advance.

Andy

nathanweeks commented 1 day ago

It sounds like you might need to allocate more memory to your Slurm job (e.g., with the sbatch --mem option), but your institution's HPC support group would probably be the best first contact point to troubleshoot job resource allocation / utilization questions. If they can't determine the root cause, you might try providing your job scripts (and if necessary input files) to see if I or another community member have any ideas.

AndyLy2Zy commented 1 day ago

I pooled 2300 protein.fa files of yeast strains to one pool_protein.fa file (~5.2G). The content of the sbatch.sh see bellow: exonerate --model p2g -q ~/data/refgenome/01_genome/02_Yeast/04_sgd/S288C_reference_genome_R64-5-1_20240529/test/prot/pool.fa -t sample1.scaffold.fasta --showtargetgff yes --showquerygff no --showalignment no > test.gff

nathanweeks commented 22 hours ago

Any Slurm job script directives (e.g., #SBATCH --mem=4g) or command-line options (e.g., sbatch --mem=4g) that determine the amount of memory allocated to the job would be important as well (or cluster defaults if any such options were omitted).

AndyLy2Zy commented 22 hours ago

Yes. I set the: #SBATCH --mem=120G

nathanweeks commented 1 hour ago

I also observed increasing memory usage (increasing to over 540G before I terminated the process) with exonerate v2.4.0 (both compiled from source & from biocontainers image) with a smaller protein query (~270MB) and a ~230KB reference. I suspect there is a memory leak in that version.

Could you try compiling exonerate from the "master" branch of this repository (see instructions in the README.md)? I observed a constant ~1.1 GB of memory usage with that version.