search

mschubert / clustermq

R package to send function calls as jobs on LSF, SGE, Slurm, PBS/Torque, or each via SSH
https://mschubert.github.io/clustermq/
Apache License 2.0
146 stars 27 forks source link

Dynamically add workers to an existing QSys object #182

Open wlandau opened 4 years ago

wlandau commented 4 years ago

The resource requirements can ebb and flow over the course of running a drake workflow, especially for cases like https://github.com/ropensci/drake/issues/1080 if there is a possibility of dynamic branching. Is it be possible to create new workers in an existing QSys object? If not, what is your opinion about such a feature?

w <- workers(2L)
w <- workers$launch_more_workers(4L)
# ...

cc @psadil

mschubert commented 4 years ago

It's not currently possible, but the worker API should (eventually) allow for it