mlr-org / parallelMap

R package to interface some popular parallelization backends with a unified interface
https://parallelmap.mlr-org.com
Other
57 stars 14 forks source link

parallelMap should handle killed processes #51

Closed mb706 closed 4 years ago

mb706 commented 7 years ago

parallelMap could, in principle, be able to handle its children suddenly disappearing. Possible reasons for this are segfaults, being killed by an impatient user, or getting automatically killed after using too much system memory. Currently this does not appear to happen. Instead, parallelMap either hangs (and waits for Ctrl-C) or returns a shorter list than expected.

As an example, using issue 1530 (which is fixed on github already, but an easy way to trigger segfaults):

> dangerous <- function(k) { try(resample(makeLearner("classif.rknn", k=k), pid.task, cv10), silent=TRUE); k }
> res <- parallelMap(dangerous, c(1, 2, 1000, 3) , impute.error = 1)                                              
Mapping in parallel: mode = multicore; cpus = 2; elements = 3.
[Resample] cross-validation iter 1: [Resample] cross-validation iter 1: [Resample] cross-validation iter 1: 
 *** caught segfault ***
address 0x55af081ff294, cause 'memory not mapped'
*** Error in `/usr/lib64/R/bin/exec/R': free(): invalid next size (normal): 0x000055ad459aa0f0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7570b)[0x7f883c3b570b]
[..........]' lots of memory dump
> res
[[1]]
[1] 1

[[2]]
[1] 2

[[3]]
[1] 3