search

snakemake / snakemake-executor-plugin-slurm

A Snakemake executor plugin for submitting jobs to a SLURM cluster
MIT License
18 stars 19 forks source link

Add seff support to evaluate resource allocation after executation #147

Open sreichl opened 2 months ago

sreichl commented 2 months ago

Employ slurm command seff for each finished job and append output to the respective slurm-job log file to get statistics on resource usage. Over time this can help to raise awareness of more appropriate resource allocation.

cmeesters commented 2 months ago

Interesting idea. It's not going to happen 1:1 because seff is just a Perl wrapper around sacct. ;-)

However, we query sacct anyway. To extend this call and report resource usage is certainly feasible. What I would not like is to see the Memory efficiency reported by seff: this is nonsense because for some application we have to reserve a full node and hardly use memory. Yet, seff will calculate the ratio between the memory of a fully reserved node and a low memory footprint. Bad idea. I would just enjoy reading the delta.

Note: I am willing to code that, will not do it because of obligations until mid-October. If you submit a PR sooner, it will be considered.

sreichl commented 2 months ago

Cool, I am happy to wait. It would be nice to have, not a must-have-now. Thanks for your considerations and the executor plugin!