search

mschubert / clustermq

R package to send function calls as jobs on LSF, SGE, Slurm, PBS/Torque, or each via SSH
https://mschubert.github.io/clustermq/
Apache License 2.0
146 stars 27 forks source link

support for tidymodels? #232

Closed nick-youngblut closed 1 year ago

nick-youngblut commented 3 years ago

It appears that tidymodels only supports parallelization via the do* packages. That includes doFuture so I was hoping that clustermq would be supported by future, but it appears that this package is all that exists, and it is just a rough developmental package.

I really prefer using clustermq versus batchtools (thanks for making a great package!), but it looks like I'm going to have to go back to batchtools in order to parallelize tidymodels on a cluster.

Are there any plans to support tidymodels via future or another method?

mschubert commented 3 years ago

At some point, there should be both a future backend (via the package that you link), but also support for using workers as a cluster object, which would both enable your use case.

However, I do unfortunately not have an ETA for this.

mschubert commented 1 year ago

Solution to this will be a consequence of #119