search

r-lib / callr

Call R from R
https://callr.r-lib.org/
Other
297 stars 36 forks source link

Spawn background processes across HPC cluster nodes #285

Open angel-bee2018 opened 1 month ago

angel-bee2018 commented 1 month ago

Good day! 😊

Is it currently possible for users to set up a callr command such as callr::r_bg to spawn background tasks across different nodes of a compute cluster? If not, could the existing workings behind callr::r_bg be by any chance easily applied across multiple nodes?

I've been trying to figure out how to make this potentially work on a large HPC cluster which runs on openmpi as the interface for inter-node communication. From the documentation, it seems like callr::r_bg is a wrapper for processx::process, which is itself based around calling system so it theoretically it should work right ...? however, I am not familiar enough with what callr::bg, for example, adds on top of the existing processx::process function that may make this harder or easier.

I would be interested to know any thoughts about this / suggested ways we can currently achieve the goal cheers!

gaborcsardi commented 1 month ago

processx::process does not use system. More importantly the IPC in processx is a Unix socket or (a Windows named pipe), so it is not going to work across machines.