search

futureverse / future.apply

:rocket: R package: future.apply - Apply Function to Elements in Parallel using Futures
https://future.apply.futureverse.org
211 stars 16 forks source link

Expose evaluator argument of future? #30

Closed kendonB closed 3 years ago

kendonB commented 6 years ago

@HenrikBengtsson would this be as simple as adding an evaluator_ii here: https://github.com/HenrikBengtsson/future.apply/blob/0d177e5e74ed848d82d1da99021aabf9855ec138/R/future_lapply.R#L259-L284 ?

If so, would you accept a PR? cc @wlandau

HenrikBengtsson commented 6 years ago

It's not clear to me whether you want to be able to use different evaluator:s for the individual futures, or one evaluator for all of them?

kendonB commented 6 years ago

Apologies. Different evaluators for the individual futures.

HenrikBengtsson commented 6 years ago

Note that the number of futures used depends on the amount of chunking - there's one future per chunk, not necessarily one future per element in X. How would you anticipate passing such a per-future argument to the functions?

HenrikBengtsson commented 6 years ago

... and maybe it's useful for me to understand the background; what's the use case where you wish to do a future_lapply(X, FUN, ...) call but have different parts processed on different backends? Do you know which parts of X you wish to process where? If so, does splitting up X into chunks and then calling future_lapply(X[part], FUN, ...) fit the needs?

wlandau commented 6 years ago

This is related to https://github.com/ropensci/drake/issues/169 and https://github.com/ropensci/drake/issues/540. In make(parallelism = "future_lapply"), drake assigns targets to workers when their dependencies become up to date and workers become available (background here). A drake user can already select the preferred worker for each target using a worker column in their drake_plan(). Heterogeneous future_lapply() workers could make this feature very useful.

wlandau commented 6 years ago

In addition, make(parallelism = "future") spins up a new future for each individual target (1:1 transient workers). Here, the user can include an optional list column of evaluators in the drake_plan(), and drake tries to use it to launch futures with those evaluators. This feature was not working last time I checked, but the framework is there.

HenrikBengtsson commented 3 years ago

This feature request is moot since a while back, because future() no longer takes an argument evaluator. That, internal argument, was deprecated in future 1.14.0 (2019-07-01), made defunct with an error in future 1.15.0 (2019-11-07). The argument was completely removed infuture 1.20.0 (2020-10-30).