Open HenrikBengtsson opened 7 years ago
resources = ~ rng & { localhost | (file("./.Rprofile"):wr:optional & folder(".future"):wr) & globals(a , b) })
localhost
!fork
rng
r >= 4.0
capabilities
: ...package:matrixStats >= 53.0
file:"/data/input.rds"
dir:"/data":w
time:00:05:00
# HPC compatiblemem:"8GB"
# HPC compatiblemem:8*GB
mem:8:GB
os:windows
global: d49b58bb32c13e383665e52df5e0d8e0
cache: d49b58bb32c13e383665e52df5e0d8e0
tag:my_data
, tag:gpu
worker:(x >= 42)
# this will be evaluated on targeted worker and must return TRUE in order for worker to be usedIf the requirements cannot be met, a FutureError will be produced by future()
.
localhost
: we need a way to query the current backend on this. If not TRUE, we will fall back to sequential
!fork
: query backend if it used forked processing. If not answering FALSE, then use sequential
Resources can be (i) required, or (ii) optional. Among optional/preferred ones, we might have different preferences, but that might be overkill.
Dynamic specifications:
parallelly::isForkNode(node)
parallelly::isLocalhostNode(node)
isForkWorker()
isForkPlan()
UPDATE: R CMD check
will not give false positive globals when formulas are used, e.g.
hello <- function() {
model <- y ~ x + 1
message("Hello world!")
invisible(model)
}
resources
for future.batchtools. Edit 2021-01-23: can we incorporate the current list syntax into the new one, e.g.
resources = list(mem="5gb")
=>
resources = ~ batchtools:list(mem="5gb"):optional
resources = ~ batchtools:list(mem="5gb"):optional
This will allow us to use both styles
resources = ~ (file:"~/abc.csv" && r >= 4.0) + batchtools:list(mem="5gb"):optional
<resource>
<type>:<resource>
<resource>:<modifier>
<type>:<resource>:<modifier>
<type>: (host|memory|file|directory|package)
<resource>: (<expr>|<string>|<integer>|<numeric>)
<modifiers>: (required|preferred)
Examples:
<resource>:
localhost
fork
<type>:<resource>:
host:localhost
parallelization:fork
parallelization:(!fork)
parallelization:!fork
memory:2.5*GiB
file:"~/abc.csv"
file:("~/abc.csv":r)
file:("~/abc.csv":w)
file:("~/abc.csv":rw)
batchtools:list(mem="5gb")
package:xgboost
package:(xgboost >= 1.2)
<type>:<resource>:<modifier>:
file:"~/abc.csv":preferred
batchtools:list(mem="5gb"):preferred
A capability is single thing that both designates a resource and authorizes some kind of access to it. http://habitatchronicles.com/2017/05/
plan(multisession, provides = ~ tag:my_data, token:gpu:5)
Add mechanism for specifying required and preferred resources / tags, e.g.
See also