That leads me to the final issue I'm facing, when foreach with a large exported object is called from within a function, the relationship between object size and worker success is altered. This behavior similarly depends on the location of the original R process.
R session on the same node as SLURM jobs
library(foreach)
library(clustermq)
clustermq::register_dopar_cmq(n_jobs = 8, memory = 50000, timeout=180, template = list(log_file= "~/%a.log"))
data <- rep(1, 100000000)
fun <- function(i, data) {
temp <- sum(data)
i
}
x = foreach(i = 1:100, .export=c("data")) %dopar% fun(i, data)
run <- function() {
x = foreach(i = 1:100, .export=c("data")) %dopar% fun(i, data)
}
run()
This runs fine for either data <- rep(1, 100000000) or data <- rep(1, 300000000). However, I did notice something unexpected when looking at the worker memory usage. From the isolated foreach call, clustermq returns the following message:
Originally reported by @benmarchi (#179)
Issue 4
That leads me to the final issue I'm facing, when
foreach
with a large exported object is called from within a function, the relationship between object size and worker success is altered. This behavior similarly depends on the location of the original R process.R session on the same node as SLURM jobs
This runs fine for either
data <- rep(1, 100000000)
ordata <- rep(1, 300000000)
. However, I did notice something unexpected when looking at the worker memory usage. From the isolatedforeach
call,clustermq
returns the following message:But, in the
run()
call the message is:So, when
foreach
is being called through a function, the memory required by each worker is reduced by half.