search

mschubert / clustermq

R package to send function calls as jobs on LSF, SGE, Slurm, PBS/Torque, or each via SSH
https://mschubert.github.io/clustermq/
Apache License 2.0
146 stars 27 forks source link

If n_jobs is set in register_dopar_cmq, it should be the return of foreach::getDoParWorkers() #177

Closed kaneplusplus closed 4 years ago

kaneplusplus commented 4 years ago

The return of foreach::getDoParWorkers() is often used to decide how to partition data structures that are going to be processed in parallel. If n_jobs is set in register_dopar_cmq, then the number of jobs should be made available through the foreach package. Currently, it always returns NA.

Admittedly, there is a slight distinction between the number of workers and the number of jobs sent to a queue. However, I don't think it'll be an issue in practice and it would maintain orthogonality between foreach and it's backend.

I can sent a pull request later today.

mschubert commented 4 years ago

Thank you for the PR.

I remember that I had something like this, but then set the workers to NA because of an issue with it (that I can't remember right now).

I'll do some more testing and see how this works!

mschubert commented 4 years ago

merged in https://github.com/mschubert/clustermq/pull/183