HenrikBengtsson
commented
6 years ago It's not a bad feature request. I've thought about removing it in the past, but there's a conflict between this and the objective of having all futures behave as similar as ever possible.
The argument for removing the check when using plan(sequential), other than that it is "useless", is that when you write code using futures, it would be nice if 'sequential' works as if you didn't use futures at all. That minimizes the friction that the user experiences. I agree, it's not fun to have to instruct users to increase the limit via options() the first thing you do. Also, the default future.globals.maxSize is somewhat arbitrary - the "correct" limit depends on a lot of things.
The argument for keeping it is that the user gets the same experience regardless of what future backend is in use. In the same vain, this should also apply for false-negative globals, i.e. if plan(multisession) would fail due to a global variable was not picked up automatically, then plan(sequential) or plan(multicore) should give the same run-time error. I've got some ideas to achieve this, but it's tricky. In the long run, I think this strategy would avoid what we see with some other parallel code/framework in R, namely that package developers on Linux/macOS test only on forked processes (e.g. registerDoParallel(ncores)) where globals is not an issue and forget about other backends (and Windows users). There are quite a few packages using foreach() %dopar% ... out there that only works with forked processes but if they'd had been forced to test with, say, PSOCK processes, then they have gone the extra mile to make sure to properly get exported globals and packages right.
I'm not saying no to your proposal/idea, but I have to think about this one. Maybe there could be a strict = TRUE/FALSE argument/option for this. This options could be forced to strict = TRUE when running R CMD check which could then lower the risk for mistakes slipping through the cracks at the same time as end users don't have worry about it.
In the meanwhile, as a package developer (assuming that's what we're talking about here), you can temporarily disable it, or increase it by some factor, yourself in 'sequential' settings, e.g.
if (inherits(plan(), "sequential")) {
oopts <- options(future.globals.maxSize = Inf)
on.exit(options(oopts))
}It's not beautiful, but is a workaround for now.
ck37
Hello,
Would it be possible to not restrict execution based on future.globals.maxSize when plan = sequential? I recently converted a method to support parallelization via futures, but if no parallelization is used (plan = sequential) users shouldn't have to increase future.globals.maxSize because no exporting of globals should happen. What do you think?
Thanks, Chris