thomvolker
commented
1 year ago Thanks for raising these points, @edbonneville!
The reason for using plan() inside the function is to make the use of futuremice() as easy as possible for those who are not too familiar with R, but I agree that it has some serious downsides in terms of flexibility. In my opinion, the best compromise would be one in which the inexperienced R user can fall back on a simple default parallel implementation that does not require any additional specifications, whereas a more advanced R user has the freedom to customize the entire process. One option might be to use the user-specified future::plan() if such a plan is specified by the user, and leaving it as is if nothing is specified (i.e., if the plan is just sequential()), using something like the following.
oplan <- future::plan()
if (is(future::plan(), "sequential")) {
plan("multisession")
}
on.exit(future::plan(oplan), add = TRUE)However, this will not solve the second point. I again agree that reproducibility is important, but I think that this will remain an issue regardless of how we adapt futuremice(). Consider for example the following example, which yields different results depending on the future strategy (which implies similar results for differences between mice() and futuremice()):
set.seed(123)
purrr::map_dbl(1:10, ~rnorm(1))
#> [1] -0.56047565 -0.23017749 1.55870831 0.07050839 0.12928774 1.71506499
#> [7] 0.46091621 -1.26506123 -0.68685285 -0.44566197
set.seed(123)
future::plan("multisession")
furrr::future_map_dbl(1:10, ~rnorm(1),
.options = furrr::furrr_options(seed = TRUE))
#> [1] -2.03403746 -0.37386282 0.21503119 -1.17845115 -1.45315445 -2.06945243
#> [7] 1.75945588 -0.06722657 0.92327056 -0.56712447Created on 2023-05-19 with reprex v2.0.2
I haven't done much research into whether it is possible to obtain exactly reproducible results regardless of the future strategy, so if you know a way to mitigate this issue I would be very interested to implement it! Given this, and because futuremice() was explicitly developed to improve the speed of mice() as much as possible, I do not see much merit in the m = 1-approach, but if there is an alternative approach that is (almost) as fast as the current implementation and yields reproducible results regardless of the cluster specification, I would love to implement it.
edbonneville
Dear
{mice}team,Thank you for your continued work on this package!
I had a couple of suggestions/questions regarding the
futuremice()function. The first is on how theplan()is currently set within the function, which not recommended as per one of the{future}vignettes. If someone is usingfuturemice()because they want to make use of parallel processing anyway, would it not be more flexible to let them set their ownplan()? Multiple arguments (e.g. future.plan, use.logical, n.core, parallelseed) would then no longer be necessary, and would allow advanced users to specify nested futures (e.g. when you want to parallelize imputations within parallelized simulation replications).The second (related) point is that I think one of the major advantages of using this future framework is that your results can be reproducible regardless of how another user decides to parallelize the same analysis (e.g. local computer vs. computing cluster, differing numbers of cores etc). In the current implementation,
futuremice()will not be able to reproduce the same set of imputations when only the number of cores differs (same dataset, number of imputations, and seed). For example:This is a side effect of
n.coredictating how themimputations are spread out across the cores. A way to make the imputations reproducible regardless ofn.coremight look something like this:Of course if you do some benchmarks,
futuremice2()will be slower thanfuturemice(), since the former callsmice()a wholemtimes while the latter onlyn.coretimes. Nevertheless, a)futuremice2()will still outperform sequentialmice()(depending on size of dataset and number of imputations); b) you could speed it up further by for example callingmice()a first time to perform the necessary checks, and then only call a function which would boil down to runningmice()from here, which uses the defined predictorMatrix/maxit/method/etc from the first call.So my main point: the current implementation does optimize computational resources nicely, but not reproducibility - so something like
futuremice2()could be a compromise. What do you think? I guess two arguments for keeping the current implementation is that a)futuremice()should focus on optimizing speed; b) we should anyway be using a large enough number of imputed datasets such that our results do not differ much between independent imputation rounds.. :sweat_smile:Thanks!
Session info