HenrikBengtsson / future.BatchJobs

:rocket: R package: future.BatchJobs: A Future API for Parallel and Distributed Processing using BatchJobs [Intentionally archived on CRAN on 2021-01-08]
https://cran.r-project.org/package=future.BatchJobs
8 stars 0 forks source link

Add argument `resources` for BatchJobsAsyncTask #48

Closed HenrikBengtsson closed 8 years ago

HenrikBengtsson commented 8 years ago

Add argument resources for BatchJobsAsyncTask futures. These resource parameters should be passed along in the submitJobs() call.

HenrikBengtsson commented 8 years ago

The following now works:

library("async")
> plan(batchjobs, backend="~/.BatchJobs.R", resources=list(nodes="1:ppn=2", vmem="1gb"))
> v %<=% { availableCores() }
> v
PBS
  2

where ~/.BatchJobs.R is:

cluster.functions <- makeClusterFunctionsTorque("~/.BatchJobs,torque.tmpl")
cluster.functions$listJobs = function(conf, reg) { runOSCommandLinux("qselect", "-u $USER -s EHQRTW")$output }
staged.queries <- TRUE # To avoid file system locks, cf. [1]

and ~/.BatchJobs,torque.tmpl is:

## Job name
#PBS -N <%= job.name %>

## Merge standard error and output:
#PBS -j oe

## Direct streams to our logfile:
#PBS -o <%= log.file %>

## Resources needed:
<% if (length(resources) > 0) {
  opts <- unlist(resources, use.names=TRUE)
  opts <- sprintf("%s=%s", names(opts), opts)
  opts <- paste(opts, collapse=",") %>
#PBS -l <%= opts %>
<% } %>

## Run R:
Rscript --verbose "<%= rscript %>"
HenrikBengtsson commented 8 years ago

With new %tweak% in future we can/will be able to also do:

> library("async")
> plan(batchjobs, backend="~/.BatchJobs.R")

> x %<=% { availableCores() }
> y %<=% { availableCores() } %tweak% list(resources=list(vmem="1gb", nodes="1:ppn=2"))

> x
PBS
  1
> y
PBS
  2
HenrikBengtsson commented 8 years ago

And with new `%resources% in async, we can do:

> library("async")
> plan(batchjobs, backend="~/.BatchJobs.R")

> x %<=% { availableCores() }
> y %<=% { availableCores() } %resources% list(vmem="1gb", nodes="1:ppn=2")

> x
PBS
  1
> y
PBS
  2