search

nachalca / example_slurmArray

0 stars 0 forks source link

Try `drake` + `future.batchtools` #1

Open wlandau-lilly opened 6 years ago

wlandau-lilly commented 6 years ago

From wlandau-lilly/drake#42:

library(future)
library(future.batchtools)
library(drake)

backend(batchtools_slurm) # same as future::plan()

# Alternatively, plan to send targets to SLURM and then give each target multiple cores.
# This should work if you use `future_lapply()` somewhere within a target's command.
# backend(list(batchtools_slurm, multicore))

# Cap the max jobs with options(mc.cores = 2) or something similar from ?future.options
load_basic_example()
make(my_plan, parallelism = "future_lapply")
wlandau-lilly commented 6 years ago

Edit: I am having more trouble than I previously anticipated. Batchtools backends require configuration template files.

backend(batchtools_slurm(template = "batchtools.slurm.tmpl"))

I need to access SLURM and learn batchtools in earnest before I can make this functionality useful. In the meantime, the specially-configured Makefile parallelism in wlandau-lilly/drake#115 may help. There is more configurability now than just a shell.sh.