search

mschubert / clustermq

R package to send function calls as jobs on LSF, SGE, Slurm, PBS/Torque, or each via SSH
https://mschubert.github.io/clustermq/
Apache License 2.0
145 stars 26 forks source link

Expose run time as an option whilst submitting a function #297

Closed nickholway closed 1 year ago

nickholway commented 1 year ago

Hi,

A lot of clusters are configured so you have to provide a maximum runtime at the time of submission. ClusterMQ lets you specify a memory request whilst submitting a function, it'd be nice to expose a similar flag for run time.

On many UGE/SGE clusters this maps to h_rt, so having something like #$ -l h_rt={{ max_runtime | 360000 }} exposed as a parameter for Q would be great

mschubert commented 1 year ago

You can already do this by using your own custom template that includes this line and then call Q with template=list(max_runtime=xxx).

See ?Q for details.

If that doesn't work for any reason please reopen.

nickholway commented 1 year ago

That's exactly what I'm doing for the in-house examples that I'm writing, I was just thinking it could be more generally useful to have this.

mschubert commented 1 year ago

Yes, there is some trade-off of what to expose directly as argument to Q and a more general mechanism that can fill in everything (including user-defined fields).

For run time, I don't have a good default so I'm not including it (and many schedulers work without specifying). However, as you say as well, including it for yourself/your users I'd think is straightforward enough.