Error in makeClusterPSOCK() on CRAN R-development Debian and Fedora environments

[idblr]

I am a maintainer for a package that Imports the future package and I have a CRAN Error that was traced to parallelly::makeClusterPSOCK()

Error: PSOCK cluster setup strategy 'parallel' is not yet supported

This error appears on CRAN R-development Debian and Fedora environments and throws no errors on other CRAN environments.

I was able to replicate the error on R-Hub:

Debian Linux, R-devel, clang, ISO-8859-15 locale (testthat.Rout.fail) Debian Linux, R-devel, GCC (testthat.Rout.fail) Fedora Linux, R-devel, clang, gfortran (testthat.Rout.fail) Fedora Linux, R-devel, GCC (testthat.Rout.fail)

And R-Hub appears to be using parallelly v1.27.0

Any insight on this error would be much appreciated, thank you!

[HenrikBengtsson]

Hi, sorry, I only had a chance to respond now. Unfortunately, R-hub has deleted those results.

A quick work-around fix should be to just add:

options(parallelly.makeNodePSOCK.setup_strategy = "sequential")

to the problematic tests. I'll report back when I have more time to troubleshoot and reproduce this one.

PS. It's an odd problem since it appears only for R-devel and I'd expect it to show up elsewhere too, e.g. my https://cran.r-project.org/web/checks/check_results_parallelly.html and https://cran.r-project.org/web/checks/check_results_future.html but also in several of the reverse dependency packages of parallelly and future.

[idblr]

Thanks, @HenrikBengtsson. Below are R-Hub tests I ran with your quick work-around, for your reference.

Debian Linux, R-devel, clang, ISO-8859-15 locale (testthat.Rout.fail) Debian Linux, R-devel, GCC (testthat.Rout.fail) Fedora Linux, R-devel, clang, gfortran (testthat.Rout.fail) Fedora Linux, R-devel, GCC (testthat.Rout.fail) Windows Server 2008 R2 SP1, R-devel, 32/64 bit (testthat.Rout.fail)

The error persists so I will skip the parallel tests on CRAN in the meantime. Let me know if you would like any more details and if I can help in any way

[HenrikBengtsson]

The error persists so I will skip the parallel tests on CRAN in the meantime. Let me know if you would like any more details and if I can help in any way

Did you try with options(parallelly.makeNodePSOCK.setup_strategy = "sequential")?

[idblr]

The most recent tests have the following in the testthat.R file based on the suggested solution for MacOS environments in #62:

if (getRversion() >= "4.0.0" && getRversion() <= "4.1.0") {
  options(parallelly.makeNodePSOCK.setup_strategy = "sequential")
}

but if I remove the if statement and just add:

options(parallelly.makeNodePSOCK.setup_strategy = "sequential")

then the testthat are clean across all R-devel environments listed above. So I will use this work-around (and not skip on CRAN)

[HenrikBengtsson]

Good to hear.

[HenrikBengtsson]

Pasting R CMD check errors from CRAN for the record and future troubleshooting:

Version: 0.2.3
Check: tests
Result: ERROR
     Running 'spelling.R' [0s/1s]
     Running 'testthat.R' [19s/24s]
    Running the tests in 'tests/testthat.R' failed.
    Complete output:
     > library(testthat)
     > library(sparrpowR)
     Loading required package: spatstat.geom
     Loading required package: spatstat.data
     spatstat.geom 2.2-2
     Loading required package: spatstat.core
     Loading required package: nlme
     Loading required package: rpart
     spatstat.core 2.3-0
     Loading required package: spatstat.linnet
     spatstat.linnet 2.3-0
     Loading required package: spatstat

     spatstat 2.2-0 (nickname: 'That's not important right now')
     For an introduction to spatstat, type 'beginner'

     Welcome to {sparrpowR} version 0.2.3
     > help("sparrpowR") # for documentation
     > citation("sparrpowR") # for how to cite

     >
     > test_check("sparrpowR")

     == Skipped tests ===============================================================
     * On CRAN (1)

     == Failed tests ================================================================
     -- Error (test-jitter_power.R:73:3): parallel processing with future package functions properly --
     Error: PSOCK cluster setup strategy 'parallel' is not yet supported
     Backtrace:
     x
     1. +-testthat::expect_named(...) test-jitter_power.R:73:2
     2. | \-testthat::quasi_label(enquo(object), label, arg = "object")
     3. | \-rlang::eval_bare(expr, quo_get_env(quo))
     4. \-sparrpowR::jitter_power(...)
     5. \-future::plan(future::multisession, workers = n_core)
     6. \-future:::plan_set(...)
     7. \-future:::plan_init()
     8. \-future:::evaluator(...)
     9. \-future:::strategy(..., envir = envir, workers = workers)
     10. \-future::MultisessionFuture(...)
     11. \-future::ClusterFuture(...)
     12. +-base::do.call(...)
     13. \-(function (future, workers = NULL, ...) ...
     14. \-future:::ClusterRegistry("start", workers = workers, ...)
     15. \-future:::makeCluster(workers, ...)
     16. \-parallelly::makeClusterPSOCK(workers, ...)
     -- Error (test-test-spatial_power.R:205:3): parallel processing with future package functions properly --
     Error: PSOCK cluster setup strategy 'parallel' is not yet supported
     Backtrace:
     x
     1. +-testthat::expect_named(...) test-test-spatial_power.R:205:2
     2. | \-testthat::quasi_label(enquo(object), label, arg = "object")
     3. | \-rlang::eval_bare(expr, quo_get_env(quo))
     4. \-sparrpowR::spatial_power(...)
     5. \-future::plan(future::multisession, workers = n_core)
     6. \-future:::plan_set(...)
     7. \-future:::plan_init()
     8. \-future:::evaluator(...)
     9. \-future:::strategy(..., envir = envir, workers = workers)
     10. \-future::MultisessionFuture(...)
     11. \-future::ClusterFuture(...)
     12. +-base::do.call(...)
     13. \-(function (future, workers = NULL, ...) ...
     14. \-future:::ClusterRegistry("start", workers = workers, ...)
     15. \-future:::makeCluster(workers, ...)
     16. \-parallelly::makeClusterPSOCK(workers, ...)

     [ FAIL 2 | WARN 0 | SKIP 1 | PASS 22 ]
     Error: Test failures
     Execution halted
Flavor: r-devel-linux-x86_64-debian-clang

[HenrikBengtsson]

I can reproduce this locally on Ubuntu 18.04 R-devel with R --vanilla CMD check sparrpowR_0.2.3.tar.gz so now I've got something to work with. It's still odd to me, because this error is not triggered when I test with parallelly or future.

[HenrikBengtsson]

I can also reproduce this on Ubuntu 18.04 with R-devel () using sparrpowR 0.2.4;

> y <- sparrpowR::spatial_power(x_case = 0.25, x_control = 0.25, y_case = 0.75, y_control = 0.75, n_case = 10, s_case = 0.05, samp_case = "MVN", samp_control = "MVN", parallel = TRUE, n_core = 2)
Error in makeClusterPSOCK(workers, ...) : 
  PSOCK cluster setup strategy 'parallel' is not yet supported
> traceback()
14: stop(ex)
13: makeClusterPSOCK(workers, ...)
12: makeCluster(workers, ...)
11: ClusterRegistry("start", workers = workers, ...)
10: (function (future, workers = NULL, ...) 
    {
        if (is.function(workers)) 
            workers <- workers()
        if (is.null(workers)) {
            getDefaultCluster <- importParallel("getDefaultCluster")
            workers <- getDefaultCluster()
            workers <- addCovrLibPath(workers)
        }
        else if (is.character(workers) || is.numeric(workers)) {
            workers <- ClusterRegistry("start", workers = workers, 
                ...)
        }
        else {
            workers <- as.cluster(workers)
            workers <- addCovrLibPath(workers)
        }
        if (!inherits(workers, "cluster")) {
            stop("Argument 'workers' is not of class 'cluster': ", 
                paste(sQuote(class(workers)), collapse = ", "))
        }
        stop_if_not(length(workers) > 0)
        name <- attr(workers, "name", exact = TRUE)
        if (is.null(name)) {
            name <- digest(workers)
            stop_if_not(length(name) > 0, nzchar(name))
            attr(workers, "name") <- name
        }
        future$workers <- workers
        future <- structure(future, class = c("ClusterFuture", class(future)))
        future
    })(<environment>, workers = 2L, user = NULL, master = NULL, revtunnel = TRUE, 
        homogeneous = NULL, rscript_libs = c("/home/hb/R/x86_64-pc-linux-gnu-library/4.2", 
        "/home/hb/software/R-devel/trunk/lib/R/library"))
9: do.call(as_ClusterFuture, args = c(list(future, workers = workers, 
       user = user, master = master, revtunnel = revtunnel, homogeneous = homogeneous), 
       args[!future_args]))
8: ClusterFuture(expr = expr, substitute = FALSE, envir = envir, 
       workers = workers, ...)
7: MultisessionFuture(..., workers = workers, lazy = lazy, rscript_libs = rscript_libs, 
       envir = envir)
6: strategy(..., envir = envir, workers = workers)
5: evaluator(NA, label = "future-plan-test", globals = FALSE, lazy = FALSE)
4: plan_init()
3: plan_set(newStack, skip = .skip, cleanup = .cleanup, init = .init)
2: future::plan(future::multisession, workers = n_core)
1: sparrpowR::spatial_power(x_case = 0.25, x_control = 0.25, y_case = 0.75, 
       y_control = 0.75, n_case = 10, s_case = 0.05, samp_case = "MVN", 
       samp_control = "MVN", parallel = TRUE, n_core = 2)

> sessionInfo()
R Under development (unstable) (2021-08-02 r80696)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.5 LTS

Matrix products: default
BLAS:   /home/hb/software/R-devel/trunk/lib/R/lib/libRblas.so
LAPACK: /home/hb/software/R-devel/trunk/lib/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] spam_2.7-0            spatstat.linnet_2.3-0 listenv_0.8.0        
 [4] splines_4.2.0         lattice_0.20-44       tcltk_4.2.0          
 [7] colorspace_2.0-2      spatstat.utils_2.2-0  vctrs_0.3.8          
[10] viridisLite_0.4.0     mgcv_1.8-36           utf8_1.2.2           
[13] rlang_0.4.11          spatstat.data_2.1-0   pillar_1.6.2         
[16] spatstat_2.2-0        glue_1.4.2            sp_1.4-5             
[19] rngtools_1.5          doRNG_1.8.2           foreach_1.5.1        
[22] lifecycle_1.0.0       spatstat.core_2.3-0   fields_12.5          
[25] dotCall64_1.0-1       munsell_0.5.0         gtable_0.3.0         
[28] raster_3.4-13         future_1.21.0-9002    codetools_0.2-18     
[31] misc3d_0.9-0          doParallel_1.0.16     parallel_4.2.0       
[34] fansi_0.5.0           Rcpp_1.0.7            tensor_1.5           
[37] scales_1.1.1          abind_1.4-5           doFuture_0.12.0      
[40] deldir_0.2-10         parallelly_1.27.0     gridExtra_2.3        
[43] sparr_2.2-15          ggplot2_3.3.5         digest_0.6.27        
[46] spatstat.sparse_2.0-0 polyclip_1.10-0       grid_4.2.0           
[49] tools_4.2.0           magrittr_2.0.1        maps_3.3.0           
[52] goftest_1.2-2         tibble_3.1.3          crayon_1.4.1         
[55] sparrpowR_0.2.4       pkgconfig_2.0.3       ellipsis_0.3.2       
[58] Matrix_1.3-4          iterators_1.0.13      viridis_0.6.1        
[61] R6_2.5.0              globals_0.14.0        rpart_4.1-15         
[64] spatstat.geom_2.2-2   nlme_3.1-152          compiler_4.2.0       

This gives me something to work with.

[HenrikBengtsson]

I've fixed this in the next version of parallelly, cf. commit f630586.

[idblr]

Great news! Thanks for the focus and rapid solution

[HenrikBengtsson]

More troubleshooting in an attempt to be able to create a parallelly package test for this one. A minimal reproducible example;

$ R --vanilla
R Under development (unstable) (2021-08-02 r80696) -- "Unsuffered Consequences"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
...
> cl <- parallelly::makeClusterPSOCK(1)

> loadNamespace("sparrpowR")   ## <= triggers the error
> cl <- parallelly::makeClusterPSOCK(1)
Error in makeClusterPSOCK(workers, ...) : 
  PSOCK cluster setup strategy 'parallel' is not yet supported

[HenrikBengtsson]

Following the package dependencies, here's an even smaller example:

$ R --quiet --vanilla
...
loadNamespace("future")
loadNamespace("sparr")
cl <- parallelly::makeClusterPSOCK(1)
# Error in makeClusterPSOCK(workers, ...) : 
#   PSOCK cluster setup strategy 'parallel' is not yet supported

Continuing down this path;

loadNamespace("future")
loadNamespace("misc3d")
cl <- parallelly::makeClusterPSOCK(1)
# Error in makeClusterPSOCK(workers, ...) : 
#   PSOCK cluster setup strategy 'parallel' is not yet supported

One more generation, and, voila, there's tcltk in the dependency graph. Now parallelly 1.27.0 tries to detect when tcltk has been loaded and adjust accordingly. However, it only does so when parallelly is loaded. But, since sparrpowR depends on future, then parallelly is loaded before tcltk is loaded. Thus, the workaround fails. Here's a minimal example:

Fails:

> loadNamespace("parallelly")
<environment: namespace:parallelly>
> loadNamespace("tcltk")
<environment: namespace:tcltk>
> cl <- parallelly::makeClusterPSOCK(1)
Error in parallelly::makeClusterPSOCK(1) : 
  PSOCK cluster setup strategy 'parallel' is not yet supported

Swapping the order, works:

> loadNamespace("tcltk")
<environment: namespace:tcltk>
> loadNamespace("parallelly")
<environment: namespace:parallelly>
> cl <- parallelly::makeClusterPSOCK(1)

I actually knew this when I implemented the workaround and meant to improve it, but forgot.

[HenrikBengtsson]

I actually knew this when I implemented the workaround and meant to improve it, but forgot.

This case in now handled by https://github.com/HenrikBengtsson/parallelly/blob/07d27d0d43b88891f878429956b75ff620351fdb/R/parallelly_disable_parallel_setup_if_needed.R#L93-L97

So, everything has been solved surrounding tcltk (and sparrpowR) in the next release of parallelly.

[HenrikBengtsson]

parallelly 1.28.1, which fixes this problem, just made it to CRAN.