search

mschubert / clustermq

R package to send function calls as jobs on LSF, SGE, Slurm, PBS/Torque, or each via SSH
https://mschubert.github.io/clustermq/
Apache License 2.0
146 stars 27 forks source link

Idea: standardized templates for backends #231

Closed nsheff closed 1 year ago

nsheff commented 3 years ago

Thanks for clustermq, I haven't used it yet but it looks very useful. I have written a package with similar goals in python called divvy.

It strikes me that it might be useful to have standard specification for back-end templates that could be used from either python or R. So, I have been looking for someone on the R side with a similar goal, and thought I'd reach out.

The idea is that you'd "configure your computing environment just once. You'd set up the template for SLURM or LFS or AWS Batch or whatever, and provide your settings like partition name, or whatever other variables you need. Then, we'd provide tools in both R and python that can use that standardized settings to submit jobs.

What I'm proposing is basically what you're already doing, except take the way you're storing the submission templates and settings and make it standard and independent of clustermq, so that one configuration can work for tools in various languages.

let me know if you're interested in pursuing that idea.

mschubert commented 1 year ago

I realize this is a very late answer: Thanks for reaching out, but I'm not currently planning this.