Closed mtmorgan closed 2 months ago
The parallel
interface currently requires synchronization of each daemon on startup for additional safety to ensure that computations can begin immediately (e.g. in a batch script). I have prioritised robustness and not looked to particularly optimise here (as daemon startup is a one-off rather than a recurring operation).
Having said that, for the same R version on Linux on x86_64 (Ubuntu jammy) there is practically no difference in speed when I test your reprex above, so I can only assume that (as you mention) this is only for 'some platforms' i.e. aarch64-apple-darwin23.5.0. I have no particular insight as to this platform.
EDIT: I've tested on Windows x86_64 and mirai is actually faster here as I guess the underlying implementation is more optimal than base R's on this platform.
It may be possible to avoid by using the native mirai interface with mirai_map()
, if that's an option.
I see that I had a naive implementation that really did do launches synchronously... So now in #124 I launch all daemons asynchronously, and synchronize afterwards.
On Linux, mirai::make_cluster(10)
now only takes 1/3 of the time of parallel::makePSOCKcluster(10)
.
Thanks @mtmorgan for pointing this out, much appreciated - this will help not just those using the parallel
interface. I expect this should also fix the slowness you were experiencing on apple-aarch64.
These are now approximately comparable on my machine
> system.time(cl <- parallel::makePSOCKcluster(10)); parallel::stopCluster(cl)
user system elapsed
0.008 0.010 0.601
> system.time(m <- mirai::make_cluster(10))
user system elapsed
0.039 0.051 0.642
Thanks for looking into this.
You're welcome. Thanks for posting an issue!
A simple example shows that
parallel::makePSOCKcluster()
is 6x faster than mirai'smake_cluster()
on some platforms.I think this is because the PSOCK processes are launched asynchronously and then collected, whereas mirai's daemons are created strictly synchronously.