API: Specifying required and preferred resources / tags

[HenrikBengtsson]

Add mechanism for specifying required and preferred resources / tags, e.g.

f <- future(..., needs = c("path:/shared/data"), prefers = c("tag:himem"))

See also

• Issue HenrikBengtsson/future-ideas#15, HenrikBengtsson/future-ideas#302
• https://stackoverflow.com/questions/45868285/cluster-job-management-in-r-via-future-package/45898907#45898907

[HenrikBengtsson]

Grammar of Resources

Examples

resources = ~ rng & { localhost | (file("./.Rprofile"):wr:optional & folder(".future"):wr) & globals(a , b) })

• localhost
• !fork
• rng
• r >= 4.0
• capabilities: ...
• env:CLUSTER %in% c("Berkeley", "UCSF")
• `option:golbat=42
• package:matrixStats >= 53.0
• file:"/data/input.rds"
• dir:"/data":w
• time:00:05:00 # HPC compatible
• mem:"8GB" # HPC compatible
• mem:8*GB
• mem:8:GB
• os:windows
• global: d49b58bb32c13e383665e52df5e0d8e0
• cache: d49b58bb32c13e383665e52df5e0d8e0
• tag:my_data, tag:gpu
• worker:(x >= 42) # this will be evaluated on targeted worker and must return TRUE in order for worker to be used

If the requirements cannot be met, a FutureError will be produced by future().

• localhost: we need a way to query the current backend on this. If not TRUE, we will fall back to sequential
• !fork: query backend if it used forked processing. If not answering FALSE, then use sequential

[HenrikBengtsson]

Resources can be (i) required, or (ii) optional. Among optional/preferred ones, we might have different preferences, but that might be overkill.

[HenrikBengtsson]

Dynamic specifications:

• file:{{ pathname }}
• memory:{{ size }}*GB

[HenrikBengtsson]

• Need a way to query backends whether or not they support the requested resources

[HenrikBengtsson]

parallelly::isForkNode(node) parallelly::isLocalhostNode(node) isForkWorker() isForkPlan()

[HenrikBengtsson]

• [x] Verify that R CMD check doesn't complain about resource terms being missing globals

UPDATE: R CMD check will not give false positive globals when formulas are used, e.g.

hello <- function() {
  model <- y ~ x + 1
  message("Hello world!")
  invisible(model)
}

[HenrikBengtsson]

• [ ] Need to rename argument resources for future.batchtools.

Edit 2021-01-23: can we incorporate the current list syntax into the new one, e.g.

resources = list(mem="5gb")

=>

resources = ~ batchtools:list(mem="5gb"):optional
resources = ~ batchtools:list(mem="5gb"):optional

This will allow us to use both styles

resources = ~ (file:"~/abc.csv" && r >= 4.0) + batchtools:list(mem="5gb"):optional

[HenrikBengtsson]

<resource>
<type>:<resource>
<resource>:<modifier>
<type>:<resource>:<modifier>

<type>: (host|memory|file|directory|package)
<resource>: (<expr>|<string>|<integer>|<numeric>)
<modifiers>: (required|preferred)

Examples:

<resource>:
localhost
fork

<type>:<resource>:
host:localhost
parallelization:fork
parallelization:(!fork)
parallelization:!fork
memory:2.5*GiB
file:"~/abc.csv"
file:("~/abc.csv":r)
file:("~/abc.csv":w)
file:("~/abc.csv":rw)
batchtools:list(mem="5gb")
package:xgboost
package:(xgboost >= 1.2)

<type>:<resource>:<modifier>:
file:"~/abc.csv":preferred
batchtools:list(mem="5gb"):preferred

[HenrikBengtsson]

A capability is single thing that both designates a resource and authorizes some kind of access to it. http://habitatchronicles.com/2017/05/

[HenrikBengtsson]

plan(multisession, provides = ~ tag:my_data, token:gpu:5)