future::multisession with parallelly 1.26.0

[mass-a]

Hello Henrik,

I have experienced a problem with future multisession since upgrading parallelly to 1.26.0.

plan(future::multisession)

hangs and then returns: Error in makeClusterPSOCK(workers, ...) : Cluster setup failed. 16 of 16 workers failed to connect.

If I downgrade to parallelly 1.25.0 the above runs fine.

Any suggestion to solving this?

> sessionInfo()
R version 4.1.0 Patched (2021-05-24 r80372)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur 10.16

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
[1] future.apply_1.7.0 future_1.21.0      parallelly_1.26.0 

loaded via a namespace (and not attached):
[1] BiocManager_1.30.15 compiler_4.1.0      parallel_4.1.0      tools_4.1.0         listenv_0.8.0       codetools_0.2-18    digest_0.6.27      
[8] globals_0.14.0      renv_0.13.2   

[HenrikBengtsson]

Thanks for reporting. I assume there might be others with this problem too, so I'd like to get to the bottom of this later. But for now, if you set:

options(parallelly.makeNodePSOCK.setup_strategy = "sequential")

does that work for you with parallelly 1.26.0? Could you please try/confirm this?

[mass-a]

Yes! after setting the option it works also with parallelly 1.26.0, thanks.

Can you explain what that does?

[HenrikBengtsson]

Great.

Can you explain what that does?

It disables the new faster setup strategy, cf. https://www.jottr.org/2021/06/10/parallelly-1.26.0/. You can also pass argument makeClusterPSOCK(..., setup_strategy = "sequential").

Are you experience the problem when running R in the terminal, in the RStudio Console, or both?

[mass-a]

Alright, got it. The fast setup improvement looks great.

I experience this issue with RStudio (1.4.1106 for macOS), running R in the terminal works fine.

[HenrikBengtsson]

Aha, so it's most likely an RStudio Console issue/bug. Probably the same problem that they patched with a workaround for now as described in https://github.com/rstudio/rstudio/issues/6692#issuecomment-785346223. As you see, if you're running RStudio Console on macOS ("Darwin") [UPDATE: apparently RStudio now does this for all OSes], then, and only then, the following is set when loading the parallel package:

parallel:::setDefaultClusterOptions(setup_strategy = "sequential") 

This will "fix" it when you're using:

cl <- makeCluster(...)

I thought about making the parallelly.makeNodePSOCK.setup_strategy option default to parallel:::defaultClusterOptions$setup_strategy, but since it was an internal function, I decided to skip it. In hindsight, I guess I shouldn't. I'll try to add it to parallelly 1.26.1. I don't know when RStudio will fix this problem, but I assume (=hope) that they'll do so at some point. What you could for now is to add the following to your ~/.Rprofile file:

## RStudio Console does not support setup_strategy="parallel" on macOS
## RStudio already avoids this for the 'parallel' package when loaded.
## Until parallelly (>= 1.26.1) is available, which will be agile to
## whatever RStudio sets, we have to disable it ourselved. The following
## will do this automagically when 'parallelly' is loaded and only if
## needed, e.g. it will _not_ disable when running R in the terminal.
setHook(packageEvent("parallelly", "onLoad"), function(pkgname, pkgpath) {
  if (Sys.getenv("RSTUDIO") == "1" && !nzchar(Sys.getenv("RSTUDIO_TERM")) &&
      packageVersion("parallelly") <= "1.26.0") {
    options(parallelly.makeNodePSOCK.setup_strategy = "sequential")
  }
})

[HenrikBengtsson]

Hi, again. In the next release, parallelly will be agile to whatever RStudio choose to do here. Until released, you can install it as:

remotes::install_github("HenrikBengtsson/parallelly@1.26.0-9001")

That should give you parallelly (>= 1.26.0-9001). When running with that, it's automatically set the above R option if needed.

If you can very that this works for you in your setup, that would be great.

[pat-s]

If you can very that this works for you in your setup, that would be great.

Works for me on macOS with R 4.1.0 and RStudio 1.5.55.

[mass-a]

If you can very that this works for you in your setup, that would be great.

The development version of parallelly works fine in my setup. Thanks again for the quick feedback and fix!

[osorensen]

I'm experiencing the same issue when using future/furrr, and posting the solution here in case anyone else experiences the same issue.

The following would cause RStudio to hang:

library(furrr)
plan(multisession)

Modifying @HenrikBengtsson's solution slightly (future.makeNodePSOCK.setup_strategy rather than parallelly.makeNodePSOCK.setup_strategy) fixed the issue:

options(future.makeNodePSOCK.setup_strategy = "sequential")
library(furrr)
plan(multisession)

[HenrikBengtsson]

Note that these future.* options are just aliases for the corresponding parallelly.* options, so the latter should definitely work.

Also, I'm hoping to submit a new version of parallelly any day now that works around this bug in R automatically. Thus, consider this option workaround a very temporary "hack".

[osorensen]

Sorry, @HenrikBengtsson, and thanks for correcting me! I was too quick, and see now that everything works with parallelly.*

[HenrikBengtsson]

Upgrading to parallelly 1.26.1, which just reached CRAN, solves this problem (by working around a bug in R, which has been fixed in the upcoming R 4.1.1).