futureverse / future.apply

:rocket: R package: future.apply - Apply Function to Elements in Parallel using Futures
https://future.apply.futureverse.org
211 stars 16 forks source link

Environment for future_lapply #70

Open dipterix opened 3 years ago

dipterix commented 3 years ago

In future, we can set environment where to search for globals, but in future_lapply, this option seems to be removed.

Is it possible to do this:

(original code)

future_lapply <- function(X, FUN, ..., future.envir2 = parent.frame()){
  ...
  values <- future_xapply(
    ...,
    future.envir2 = future.envir2
  )
}

In this way, future_xapply will be able to receive environment in case other Future classes need to process the environment.

https://github.com/HenrikBengtsson/future.apply/blob/fcec8077a4eb8da0b4e60dd67a20fca33715942f/R/future_xapply.R#L186-L195

We only need to change line 188 to

fs[[ii]] <- future(
      ...
      envir = future.envir2,
      ...
)
HenrikBengtsson commented 3 years ago

Yes, this is a known issue and on the road map (for way too long), e.g. Issue #62 and https://github.com/HenrikBengtsson/future.apply/issues/64#issuecomment-733487812. I just need some deep focus to get this correct; passing down envir = parent.frame() as-is will result in some revdep complaints related to globals and ... that need to be understood and sorted out.

HenrikBengtsson commented 3 years ago

Also, I was hoping to have gotten around and created https://github.com/HenrikBengtsson/future.mapreduce by now so it can just implemented it there and then future.apply, doFuture, and furrr could depend on that.

dipterix commented 3 years ago

Sounds good!