pblapply() silently produces NULL while lapply() succeeds

[dashaub]

Apologies that this is neither minimal nor fully reproducible, but it is at least an example.

Setup

dat.gz This is running on R 3.4.4 (built from source) inside a Docker container on Debian Stretch from the Rocker images:

> sessionInfo()
R version 3.4.4 (2018-03-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 9 (stretch)

Matrix products: default
BLAS: /usr/lib/openblas-base/libblas.so.3
LAPACK: /usr/lib/libopenblasp-r0.2.19.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.10.4-3   pbapply_1.3-4         forecastHybrid_2.2.12
[4] thief_0.3             forecast_8.3         

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.16      magrittr_1.5      doParallel_1.0.11 uroot_2.0-9      
 [5] munsell_0.4.3     colorspace_1.3-2  lattice_0.20-35   rlang_0.2.0      
 [9] quadprog_1.5-5    foreach_1.4.4     TTR_0.23-3        plyr_1.8.4       
[13] tools_3.4.4       xts_0.10-2        nnet_7.3-12       parallel_3.4.4   
[17] quantmod_0.4-12   grid_3.4.4        nlme_3.1-137      timeDate_3043.102
[21] gtable_0.2.0      urca_1.3-0        iterators_1.0.9   tseries_0.10-43  
[25] lazyeval_0.2.1    lmtest_0.9-36     tibble_1.4.2      ggplot2_2.2.1    
[29] codetools_0.2-15  curl_3.2          fracdiff_1.4-2    compiler_3.4.4   
[33] pillar_1.2.1      scales_0.5.0      zoo_1.8-1   

I was also not able to reproduce this behavior with R 3.4.4 on OSX, and I've tried setting numCores <- 1 inside the Docker container and everything seems to work. Here are the minimal sets of packages necessary to reproduce the bug:

devtools::install_github("robjhyndman/forecast", ref = "0b10c5327cbbdadeb7fee23c44b1278b68f078de")
devtools::install_github("ellisp/forecastHybrid/pkg", ref = "c928178a43ed43d457d86230a2edd8f3b57cf61c")
load("dat.gz")
seriesHorizon <- 18
library(forecastHybrid)
library(pbapply)

> names(dat)
 [1] "M19567" "M45556" "M12971" "M24651" "M46624" "M17619" "M14794" "M1631" 
 [9] "M31812" "M43884" "M2144"  "M9584"  "M35428" "M6221"  "M33777" "M47591"
[17] "M44973" "M28249" "M34842" "M23189"

Bug report

The seeds really should not be necessary since all of the functions are deterministic, but I set them anyway. A regular for loop produces expected results where all objects are of class "forecast":

> set.seed(40)
> forecasts <- list()
> for(i in seq_along(dat)){
      forecasts[[i]] <- forecast(hybridModel(y = x, models = "aft", verbose = FALSE),
                                 h = seriesHorizon, level = 95,
                                 PI.combination = "mean")
 }
> table(sapply(forecasts, class))

forecast 
      20 
> which(sapply(forecasts, is.null))
integer(0)

Everything still works when using lapply:

> set.seed(40)

> forecasts <- lapply(dat, FUN = function(x) forecast(hybridModel(y = x, models = "aft",
                                                                  verbose = FALSE),
                                                      h = seriesHorizon, level = 95,
                                                      PI.combination = "mean"))
> table(sapply(forecasts, class))

forecast 
      20 
> which(sapply(forecasts, is.null))
named integer(0)

However, with pblapply() some of the jobs fail silently.

> set.seed(40)
> numCores <- 6
> forecasts <- pblapply(dat,
                        FUN = function(x) forecast(hybridModel(y = x, models = "aft",
                                                               verbose = FALSE),
                                                   h = seriesHorizon, level = 95,
                                                   PI.combination = "mean"),
                        cl = numCores)

> table(sapply(forecasts, class))

forecast     NULL 
      19        1 
> which(sapply(forecasts, is.null))
M17619 
     6 

And once more. There are failures again, but they do not reproduce the earlier results.

set.seed(40)
numCores <- 6
forecasts <- pblapply(dat,
                      FUN = function(x) forecast(hybridModel(y = x, models = "aft",
                                                             verbose = FALSE),
                                                 h = seriesHorizon, level = 95,
                                                 PI.combination = "mean"),
                      cl = numCores)
> table(sapply(forecasts, class))

forecast     NULL 
      18        2 
> which(sapply(forecasts, is.null))
M17619 M35428 
     6     13 

I can keep running the process with pblapply() and get different failures each time.

[psolymos]

Can you try with Socket cluster? It worked on Windows with R 3.3.3:

library(parallel)
cl <- makeCluster(6)
clusterEvalQ(cl,library(forecastHybrid))
clusterEvalQ(cl,seriesHorizon <- 18)
forecasts <- pblapply(dat,
                        FUN = function(x) forecast(hybridModel(y = x, models = "aft",
                                                   verbose = FALSE),
                                                   h = seriesHorizon, level = 95,
                                                   PI.combination = "mean"),
                        cl = cl)
stopCluster(cl)
table(sapply(forecasts, class))
# forecast 
#      20 

Or see if parallel::mclapply gives you the same issue:

numCores <- 6
forecasts <- parallel::mclapply(dat,
                      FUN = function(x) forecast(hybridModel(y = x, models = "aft",
                                                 verbose = FALSE),
                                                 h = seriesHorizon, level = 95,
                                                 PI.combination = "mean"),
                      mc.cores = numCores)

[dashaub]

Good debugging advice.

Everything works fine with the socket cluster approach.

When using mclapply(), I get errors when I use numCores > 1, but it works fine when numCores <- 1

So it looks like there must be something going on with the "parallel" package breaking forking inside the Docker container.

[psolymos]

Good to know it is a Docker/forking issue. A quick web search gave several hits with strange behaviour, but nothing too related to this. Something to watch for!