Determine dynamic resources depending on retry reason?

[nikostr]

Currently it is possible to adjust resources based on which attempt of a job is being run. Jobs fail for different reasons - timing out requires a different solution to out of memory errors. A neat feature would be allowing to adjust resources based on if previous attempts failed with TIMEOUT or OUT_OF_MEMORY.

[cmeesters]

Indeed. Yet: Whilst OUT_OF_MEMORY seems easy to accomplish (just allow for more), it is not that easy because what is a sensible increment? E.g. for whole node jobs, it might be sensible to require the next bigger node type (memory-wise) in the cluster. Which is that, and how can Snakemake get this info?

For TIMEOUT the difficulty is elsewhere. It can be caused by

• not making use of the fs-plugin, hence I/O contention
• asking for too few threads
• inappropriate parameterization of an application
• in the case of user code it might be unoptimized, too

And then, such messages can be misleading: SLURM occasionally does not recognize node failures but reports time-outs or similar. How do we discriminate against these reasons here?

I am all open for suggestions here. Anything which can be a simple improvement is most welcome!

[nikostr]

Regarding increments, I think it makes sense for the user to determine this? One idea: introducing the variable retry_reason to users for the dynamic resource allocation could allow for something like

constraint: 'fat' if retry_reason=="oom" else ''

in the workflow profile. Perhaps this would require updates to snakemake as a whole rather than just this plugin? Anyway, then users can design the retry logic based on whichever system they are dealing with.

Right now what I've been doing is running my pipeline once, manually re-running it with a longer requested run-time to catch the maybe 5% TIMEOUT jobs, and manually re-running it once again following this requesting large memory nodes to get the handful of memory intensive jobs sorted. It's a very okay workflow, but it would be neater if this logic could live in my workflow profile instead, haha. But yeah, maybe introducing this is just a matter of added complexity that won't save people time relative to what's needed to implement it and maintain it in workflows.

[cmeesters]

Hm, worth a thought!

That

... once, manually re-running ... to catch the maybe 5% TIMEOUT jobs, and manually re-running it once again ...

is, of course, unacceptable. The question is, though, whether there are issues at hand which can be prevented. It would merit further investigation.

[blaiseli]

I too would be interested in having access to a retry_reason variable in functions that I use to adapt resources to the situation encountered during previous retries. I can increase memory or runtime based on number of attempts, but it is a waste of resources (and time spent in the queue) if my job asks for more memory after a TIMEOUT.