MultisessionFuture () failed to call gassign() on cluster ......

[elgabbas]

Hello,

I am trying to use the furrr package, but I think I am having a problem related to the use of future.

First, when I prepare to run the script in parallel using the following command, I receive these warnings. I am unsure if I should ignore them or there is an issue I need to solve.

plan(multisession, workers = 20)

During startup - Warning messages: 1: package ‘utils’ in options("defaultPackages") was not found 2: package ‘stats’ in options("defaultPackages") was not found During startup - Warning messages: 1: package ‘utils’ in options("defaultPackages") was not found 2: package ‘stats’ in options("defaultPackages") was not found During startup - Warning messages: 1: package ‘utils’ in options("defaultPackages") was not found During startup - 2: package ‘stats’ in options("defaultPackages") was not found Warning messages: 1: package ‘utils’ in options("defaultPackages") was not found 2: package ‘stats’ in options("defaultPackages") was not found During startup - Warning messages: 1: package ‘utils’ in options("defaultPackages") was not found 2: package ‘stats’ in options("defaultPackages") was not found During startup - Warning messages: 1: package ‘utils’ in options("defaultPackages") was not found 2: package ‘stats’ in options("defaultPackages") was not found During startup - Warning message: package ‘utils’ in options("defaultPackages") was not found During startup - Warning messages: 1: package ‘utils’ in options("defaultPackages") was not found 2: package ‘stats’ in options("defaultPackages") was not found During startup - Warning message: package ‘utils’ in options("defaultPackages") was not found During startup - Warning message: package ‘utils’ in options("defaultPackages") was not found During startup - Warning message: package ‘utils’ in options("defaultPackages") was not found During startup - Warning message: package ‘utils’ in options("defaultPackages") was not found Error: Initialization of plan() failed, because the test future used for validation failed. The reason was: Package 'future' is not installed on worker (r_version: 4.1.2 (2021-11-01); platform: x86_64-conda-linux-gnu (64-bit); os: Linux 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018; hostname: prod-0316)

Then, when I use map function from the furrr package, I receive the following error:

Error: Problem with mutate() column Preds. Preds = future_map(...). MultisessionFuture () failed to call gassign() on cluster RichSOCKnode #2 (PID 114424 on ‘localhost’). The reason reported was ‘error reading from connection’. Post-mortem diagnostic: The total size of the 15 globals exported is 14.99 MiB. The three largest globals are ‘...furrr_chunk_args’ (14.35 MiB of class ‘list’), ‘Models_WS’ (374.44 KiB of class ‘list’) and ‘predictMaxEnt’ (185.81 KiB of class ‘function’) Backtrace:

1. ├─global::ExtractPreds(Data = CurrD)
2. │ └─%>%(...)
3. ├─tidyr::unnest(., cols = c(data))
4. ├─dplyr::rename_at(...)
5. │ └─dplyr:::tbl_at_vars(.tbl, .vars, .include_group_vars = TRUE)
6. │ └─dplyr::tbl_vars(tbl)
7. │ ├─dplyr:::new_sel_vars(tbl_vars_dispatch(x), group_vars(x))
8. │ │ └─base::structure(...)
9. │ └─dplyr:::tbl_vars_dispatch(x)
   10. ├─tidyr::unnest(., cols = c(Preds))
   11. ├─dplyr::mutate(...)
   12. ├─dplyr:::mutate.data.frame(...)
   13. │ └─dplyr:::mutate_cols(.data, ..., caller_env = caller_env())
   14. │ ├─base::withCallingHandlers(...)
   15. │ └─mask$eval_all_mutate(quo)
   16. └─furrr::future_map(...)
   17. └─furrr:::furrr_map_template(...)
   18. └─furrr:::furrr_template(...)
   19. └─future::future(...)
   20. ├─future::run(future)
   21. └─future:::run.Future(future)
   22. ├─future::run(future)
   23. └─future:::run.ClusterFuture(future)
   24. ├─base::suppressWarnings(...)
   25. │ └─base::withCallingHandlers(...)
   26. └─future:::cluster_call(...) Execution halted Warning message: system call failed: Cannot allocate memory

Please note that I am running this code on an HPC computer (CentOS Linux 7 ), with a total of 110 GB ram and 36 nodes. I used conda to install R (v 4.1.2). The same code worked nicely on different R installations but failed on this computer. Here is the results of the session info:

sessionInfo()

R version 4.1.2 (2021-11-01) Platform: x86_64-conda-linux-gnu (64-bit) Running under: CentOS Linux 7 (Core) Matrix products: default BLAS/LAPACK: /home/----/-----/.conda/envs/REnv2/lib/libopenblasp-r0.3.18.so locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] parallel stats graphics grDevices utils datasets methods base

other attached packages: [1] tibble_3.1.6 readr_2.1.2 furrr_0.2.3 future_1.24.0 purrr_0.3.4 dismo_1.3-5 raster_3.5-15 sp_1.4-6 sf_1.0-6 stringr_1.4.0 [11] tidyr_1.2.0 dplyr_1.0.7 snow_0.4-4

loaded via a namespace (and not attached): [1] Rcpp_1.0.8 pillar_1.7.0 compiler_4.1.2 class_7.3-20 tools_4.1.2 digest_0.6.29 lifecycle_1.0.1 lattice_0.20-45 [9] pkgconfig_2.0.3 rlang_0.4.12 cli_3.2.0 DBI_1.1.2 e1071_1.7-9 terra_1.5-21 hms_1.1.1 globals_0.14.0 [17] generics_0.1.2 vctrs_0.3.8 classInt_0.4-3 grid_4.1.2 tidyselect_1.1.1 glue_1.6.2 listenv_0.8.0 R6_2.5.1 [25] parallelly_1.30.0 fansi_1.0.2 tzdb_0.2.0 magrittr_2.0.2 codetools_0.2-18 ellipsis_0.3.2 units_0.8-0 assertthat_0.2.1 [33] utf8_1.2.2 KernSmooth_2.23-20 stringi_1.7.6 proxy_0.4-26 crayon_1.5.0

The same issue occurred when using other strategies (multisession, multicore, cluster). I do not think this is due to a memory issue because the same code runs well on the same node specifications, but using an older version of R (not installed via conda). Is there something related to using conda?

Any suggestions?

Thanks in advance, Ahmed

[orozcoae89]

Can you test this code? Is posible that you need the cl object, with follow:

library(future)
workers <-rep("localhost",4)
cl <- makeClusterPSOCK(workers, revtunnel=TRUE, outfile="")
# starting worker pid=134xx on localhost:11xxx at 13:46:34.292
# starting worker pid=134xx on localhost:11xxx at 13:46:34.580
# starting worker pid=134xx on localhost:11xxx at 13:46:34.817
# starting worker pid=134xx on localhost:11xxx at 13:46:35.050
plan <- plan(list(future::tweak(cluster, workers=workers), multisession))
plan

If you dont't solve send us the short example for replicate and help you.

Erick

[elgabbas]

Hello,

Sometimes, I also receive this error

Fatal error: couldn't allocate memory for pointer stack Fatal error: couldn't allocate memory for pointer stack Error: cannot allocate vector of size 9 Kb

caught segfault address (nil), cause 'memory not mapped' An irrecoverable exception occurred. R is aborting now ...

caught segfault address (nil), cause 'memory not mapped' An irrecoverable exception occurred. R is aborting now ...

caught segfault address (nil), cause 'memory not mapped' An irrecoverable exception occurred. R is aborting now ...

caught segfault address (nil), cause 'memory not mapped' An irrecoverable exception occurred. R is aborting now ...

caught segfault address (nil), cause 'memory not mapped' Error: memory exhausted (limit reached?) Execution halted Error: memory exhausted (limit reached?) Execution halted Warning message: system call failed: Cannot allocate memory

caught segfault address 0x5555559b9568, cause 'invalid permissions' An irrecoverable exception occurred. R is aborting now ...

Thanks, Ahmed

[elgabbas]

Thanks @orozcoae89 for your reply. Here is the results of the script you provided.

workers <-rep("localhost",4) cl <- makeClusterPSOCK(workers, revtunnel=TRUE, outfile="")

starting worker pid=175624 on localhost:11798 at 13:59:27.898 starting worker pid=175627 on localhost:11798 at 13:59:27.898 starting worker pid=175625 on localhost:11798 at 13:59:27.899 starting worker pid=175626 on localhost:11798 at 13:59:27.907

starting worker pid=134xx on localhost:11xxx at 13:46:34.292

starting worker pid=134xx on localhost:11xxx at 13:46:34.580

starting worker pid=134xx on localhost:11xxx at 13:46:34.817

starting worker pid=134xx on localhost:11xxx at 13:46:35.050

plan <- plan(list(future::tweak(cluster, workers=workers), multisession)) plan

List of future strategies:

1. sequential:
   • args: function (..., envir = parent.frame())
   • tweaked: FALSE
   • call: NULL

I will implement my code then and see if it works.

Cheers, Ahmed

[orozcoae89]

This error is common when you don't have set the connections with remote machines. Can you give us more details about your code for help you?

Erick

[elgabbas]

After running my script, I receive similar error:

Error: Problem with mutate() column Preds. ℹ Preds = future_map(...). ✖ ClusterFuture () failed to call gassign() on cluster RichSOCKnode #3 (PID 175879 on ‘localhost’). The reason reported was ‘error writing to connection’. Post-mortem diagnostic: The total size of the 15 globals exported is 72.55 MiB. The three largest globals are ‘...furrr_chunk_args’ (71.76 MiB of class ‘list’), ‘Models_WS’ (374.44 KiB of class ‘list’) and ‘predictMaxEnt’ (331.60 KiB of class ‘function’)

My code is simple. I am using future_map function from the furrr package to mutate a new column. However, the data is huge and consists of ~1M observations. Nevertheless, the same script worked nicely using the RStudio server on the HPC (same specifications), but now I need to use the same script over different datasets and therefore need to batch the jobs using different R runs .

Cheers, Ahmed

[orozcoae89]

Well, I think that you could improve your code, because this error is that you have a excedence in the globals fixed by default. More details: https://www.rdocumentation.org/packages/future/versions/1.24.0/topics/future.options. Prove this solution: options(future.globals.maxSize = ??)