Is it currently possible for users to set up a callr command such as callr::r_bg to spawn background tasks across different nodes of a compute cluster?
If not, could the existing workings behind callr::r_bg be by any chance easily applied across multiple nodes?
I've been trying to figure out how to make this potentially work on a large HPC cluster which runs on openmpi as the interface for inter-node communication. From the documentation, it seems like callr::r_bg is a wrapper for processx::process, which is itself based around calling system so it theoretically it should work right ...? however, I am not familiar enough with what callr::bg, for example, adds on top of the existing processx::process function that may make this harder or easier.
I would be interested to know any thoughts about this / suggested ways we can currently achieve the goal
cheers!
processx::process does not use system. More importantly the IPC in processx is a Unix socket or (a Windows named pipe), so it is not going to work across machines.
Good day! 😊
Is it currently possible for users to set up a
callr
command such ascallr::r_bg
to spawn background tasks across different nodes of a compute cluster? If not, could the existing workings behindcallr::r_bg
be by any chance easily applied across multiple nodes?I've been trying to figure out how to make this potentially work on a large HPC cluster which runs on
openmpi
as the interface for inter-node communication. From the documentation, it seems likecallr::r_bg
is a wrapper forprocessx::process
, which is itself based around callingsystem
so it theoretically it should work right ...? however, I am not familiar enough with whatcallr::bg
, for example, adds on top of the existingprocessx::process
function that may make this harder or easier.I would be interested to know any thoughts about this / suggested ways we can currently achieve the goal cheers!