HenrikBengtsson / parallelly

R package: parallelly - Enhancing the 'parallel' Package
https://parallelly.futureverse.org
130 stars 7 forks source link

makeClusterPSOCK(): Protect against creating too many local workers #107

Closed HenrikBengtsson closed 9 months ago

HenrikBengtsson commented 1 year ago

(This was triggered by https://stackoverflow.com/questions/77244972)

There's nothing that prevents us from doing:

cl <- makeClusterPSOCK(n)

where n is very large, e.g. n = 1000.

To protect against this, makeClusterPSOCK() could compare n > availableCores() and:

If an error, then there should be an option to override it, e.g. force = TRUE. Of course, one can always workaround it by merging many smaller clusters, e.g.

cl <- c(makeClusterPSOCK(8), makeClusterPSOCK(8))
HenrikBengtsson commented 9 months ago

Implemented and now documented in ?makeClusterPSOCK (https://parallelly.futureverse.org/reference/makeClusterPSOCK.html).