Spread the load acros two mirrored remote SLURM systems

[stemangiola]

Prework

• [ x] Read and agree to the Contributor Code of Conduct and contributing guidelines.
• [x ] If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
• [x ] New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a GitHub discussion.
• [ x] Format your code according to the tidyverse style guide.

Proposal

I am not sure if this is already possible, but let's say I have two remote SLURM clusters that have mirrored file systems (same directories and file locations). Would it be possible to spread the pipeline load across those two URLs?

Thanks a lot.

[wlandau]

You could set up a controller for each cluster, then combine them in a controller group: https://wlandau.github.io/crew/articles/groups.html. You will still need to manually decide which controller to send each task, but the interface is a bit more seamless. If you want to make a decision on the fly, you can use the saturated() method to check if a controller's workers are all busy: https://wlandau.github.io/crew/reference/crew_class_controller_group.html#method-crew_class_controller_group-saturated

[wlandau]

Converting this issue to a discussion.