rscript object missing under nested future topology (login->cluster->session)

[coatless]

I'm trying to do a nested future topology with torque cluster. However, when I run the code I'm getting the error of:

Error: Error brewing template: Error in cat(rscript) : object 'rscript' not found

I've modified the example here:

https://cran.r-project.org/web/packages/future/vignettes/future-3-topologies.html

With information given in the future.batchtools vignette:

https://cran.r-project.org/web/packages/future.batchtools/vignettes/future.batchtools.html

In particular, I'm using:

library("future")
library("future.batchtools")
library("listenv")

## Specify future topology
## login node -> { cluster nodes } -> { multiple cores }
plan(
  list(
    tweak(remote, workers = c("username@cluster.higherlearning.edu")),
    batchtools_torque,
    multiprocess
    )
)

## (a) This will be evaluated on the cluster login computer
x %<-% {
  thost <- Sys.info()[["nodename"]]
  tpid <- Sys.getpid()
  y <- listenv()
  for (task in 1:4) {

    ## (b) This will be evaluated on a compute node on the cluster
    y[[task]] %<-% {
      mhost <- Sys.info()[["nodename"]]
      mpid <- Sys.getpid()
      z <- listenv()
      for (jj in seq_len(availableCores())) {
        ## (c) These will be evaluated in separate processes on the same compute node
        z[[jj]] %<-% data.frame(task = task,
                                top.host = thost, top.pid = tpid,
                                mid.host = mhost, mid.pid = mpid,
                                host = Sys.info()[["nodename"]],
                                pid = Sys.getpid())
      }
      Reduce(rbind, z)
    }
  }
  Reduce(rbind, y)
}

# Check to see if the job is done and display values if so:
if( resolved(futureOf(x)) ) print(x)

I've used the template ~/.batchtools.torque.tmpl:

## Job name:
#PBS -N <%= job.name %>

## Merge standard error and output:
#PBS -j oe

## Resource parameters:
<% for (name in names(resources)) { %>
#PBS -l <%= name %>=<%= resources[[name]] %>
<% } %>

## Run R:
R CMD BATCH --no-save --no-restore "<%= rscript %>" /dev/stdout

Within the last line, I don't think that rscript variable is being set. Is future generating an R script with the topology and then executing? This is also different from the .pbs template configuration shipping with batchtools:

https://github.com/mllg/batchtools/blob/fc896b349e86456506387b1a5cabba7e09f1a09f/inst/templates/torque-lido.tmpl#L76-L77

[HenrikBengtsson]

Oh my. Thanks for spotting this and reporting here.

This looks like a cut'n'paste that originates from future.BatchJobs. I've updated the vignette + the README in the develop branch (=next release):

https://github.com/HenrikBengtsson/future.batchtools/tree/develop

[coatless]

@HenrikBengtsson thanks for the quick patch!

Two quick follow-up questions, following the batchtools template:

1. The only way to pass down configuration options is to use resources parameter as the ... aren't picked up? Or, is it possible to set a configuration within conf in BatchtoolsFuture()?
2. Where can I find a list of the pre-defined variables used in the updated template, e.g. log.file, job.name, et cetera?

[HenrikBengtsson]

1. Correct, resources is how you pass options to the batchtools template. This is by design of the batchtools package, and specifically batchtools::submitJobs().

2. Those are also by design of the batchtools package.