HenrikBengtsson
commented
4 years ago So, several things go on here. The underlying issue is that foreach left it to the backends to handle certain things, including parallel RNG. This is very unfortunate because it is really hard to write foreach code that works as-is with different foreach adapters. Ideally, foreach would have implemented parallel RNG itself such that it would work the same everywhere. (This is the approach I'm taking in the future framework, e.g. future.apply). Such a discussion is probably better suited over at https://github.com/RevolutionAnalytics/foreach/issues.
I did consider having doFuture have built-in support for parallel RNG much like future.apply. If I did (or end up doing), then you could do:
library(foreach)
doFuture::registerDoFuture()
set.seed(123)
y <- foreach(i = rep(3, 2)) %dopar% {
runif(i)
}and you'd get numerically identical results regardless of future-backend used (as you hoped for in Example 1). Maybe I should just add that - it's pretty easy to do. The downside is that the same foreach() will not use proper parallel RNG if you use:
doParallel::registerDoParallel(4L)instead. Currently, all existing foreach adapters requires doRNG for parallel RNG. So, you need to do something like you propose in Example 2:
doParallel::registerDoParallel(4L)
doRNG::registerDoRNG(123)Using doRNG is basically a de-facto standard (= the only solution available) that I didn't want to break. At least, I didn't want to break it or introduce yet another standard, without carefully consider it's consequences.
Howeber, maybe one can argue that if
doParallel::registerDoParallel(4L)
doRNG::registerDoRNG()remain in control of the end user, and not the developer, then it would be straightforward for the user to switch to:
future::plan("multisession", workers = 4L)
doFuture::registerDoFuture()and
y <- foreach(i = rep(3, 2)) %dopar% {
runif(i)
}would work in both cases.
I'll try to think about this more but right now I tempted to add built-in support for parallel RNG to doFuture. It would certainly lower friction for both developers and end users.
pat-s
Continuing our mail discussion here.
The variety of options, especially when using the
foreachframework seems overhwelming.Is is expected that "Example 1" and "Example 2" work in the same way even though the foreach operators are different?
Is "Example 2" the canonical way from your point of view right now?
Example 1
``` r library("doFuture") #> Loading required package: globals #> Loading required package: future #> Loading required package: foreach #> Loading required package: iterators #> Loading required package: parallel # register parallel backend registerDoFuture() plan(multisession, workers = 2) set.seed(123, "L'Ecuyer-CMRG") r15 = foreach(i = rep(3, 2)) %dopar% { runif(i) } set.seed(123, "L'Ecuyer-CMRG") r16 = foreach(i = rep(3, 2)) %dopar% { runif(i) } all.equal(r15, r16) #> [1] "Component 1: Mean relative difference: 0.2142185" #> [2] "Component 2: Mean relative difference: 0.5201712" ``` Created on 2019-12-25 by the [reprex package](https://reprex.tidyverse.org) (v0.3.0)Example 2
``` r library("doFuture") #> Loading required package: globals #> Loading required package: future #> Loading required package: foreach #> Loading required package: iterators #> Loading required package: parallel library("doRNG") #> Loading required package: rngtools #> Loading required package: pkgmaker #> Loading required package: registry #> #> Attaching package: 'pkgmaker' #> The following object is masked from 'package:base': #> #> isFALSE # register parallel backend registerDoFuture() plan(multisession, workers = 2) registerDoRNG(123) r15a = foreach(i = rep(3, 2)) %dopar% { runif(i) } registerDoRNG(123) r16a = foreach(i = rep(3, 2)) %dopar% { runif(i) } all.equal(r15a, r16a) #> [1] TRUE ``` Created on 2019-12-25 by the [reprex package](https://reprex.tidyverse.org) (v0.3.0)