search

HenrikBengtsson / future.apply

:rocket: R package: future.apply - Apply Function to Elements in Parallel using Futures
https://future.apply.futureverse.org
209 stars 16 forks source link

Distribute different tasks to different types of compute resources #2

Open wlandau-lilly opened 6 years ago

wlandau-lilly commented 6 years ago

Migrated from HenrikBengtsson/future#181, would support wlandau-lilly/drake#169. I would like to have multiple heterogeneous plans()'s and distribute different tasks to different plan()s.

library(future.batchtools)
plans <- list(plan1 = plan(multicore), plan2 = plan(batchtools_slurm(...)))
which_plans <- rep(c("plan1", "plan2"), each = 5)
future_lapply(
  X = 1:10,
  FUN = sqrt,
  plans = plans,
  which_plans = which_plans
)

I tried with ordinary futures outside of future_lapply(), but the distributed parallelism on SGE did not work.

kendonB commented 5 years ago

@wlandau This seems like the simple solution: https://github.com/HenrikBengtsson/future.apply/issues/30

recall that R lists store pointers when there are repeated objects so memory isn't an issue here. And the return value of plan is the previous plan except if it's the first plan. So you'd need to do something like this: plan(sequential); local <- plan(sequential) plan(multicore); distributed <- plan(multicore)