search

mschubert / clustermq

R package to send function calls as jobs on LSF, SGE, Slurm, PBS/Torque, or each via SSH
https://mschubert.github.io/clustermq/
Apache License 2.0
146 stars 27 forks source link

Multiple SSH hosts #217

Open hongyuanjia opened 3 years ago

hongyuanjia commented 3 years ago

I am new to cluster computing. I successfully setup clustermq using SSH. Forgive me if this question is silly, is it possible to utilize multiple ssh hosts?

I tried directly append the clustermq.ssh.host option, but obviously it did not work:

options(
    clustermq.scheduler = "ssh",
    clustermq.ssh.host = c("user@host1", "user@host2"), 
    clustermq.ssh.log = "~/cmq_ssh.log"
)
hongyuanjia commented 3 years ago

Just found https://github.com/mschubert/clustermq/issues/102#issuecomment-419657252. Hopefully this feature can be implemented in the near future.

mschubert commented 3 years ago

We should eventually support that, but it will still take some time.

If I understand correctly, parallelly cal already do this, you might want to check it out instead.

hongyuanjia commented 3 years ago

Thanks for the info! I find parallelly package quite useful. I am able to set up a cluster using it. Hope multiple SSH hosts can be supported in the near future of clustermq.