doJobCollection error

[mkrzak]

Hi, I am running future.batchtools with on LSF cluster and getting error:

Error : BatchtoolsFutureError for BatchtoolsFuture ('future_lapply-1') captured by batchtools: ‘Failed to subset JobCollection using array environment variable 'NA' [='NA']’

I am running it using "Rscript run.R" on terminal where run.R contains:

library(doFuture) library(future.batchtools)

myProc <- function(size=10000000) { vec <- rnorm(size) return(sum(vec)) }

future::plan( future.batchtools::batchtools_lsf, template="~/repos/myrepos/parallel-computing/batchtools/lsf.tmpl", resources=list( queue="normal", walltime=60,#minutes memory=8192, ncpus=2, name="merging", output="out.txt", chunks.as.arrayjobs=TRUE))

ha = future.apply::future_lapply(1:10, function(n){myProc(n)})

Additionally I have lsf.tmpl file with:

BSUB -J <%= resources$name %> # Name of the job

BSUB -o <%= resources$output %> # Output is sent to logfile, stdout + stderr > by default

BSUB -q <%= resources$queue %> # Job queue

BSUB -W <%= round(resources$walltime / 60, 1) %> # Walltime (LSF requires minutes, > batchtools uses seconds)

BSUB -M <%= resources$memory %> # Memory requirements, e.g. "5000KB", > "500MB", "5GB" etc.

export DEBUGME=<%= Sys.getenv("DEBUGME") %> <%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%> <%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%> <%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%> Rscript -e 'batchtools::doJobCollection("<%= uri %>")'

And batchtools.conf.R file with:

default.resources=list(walltime=60, memory=1024, ncpus=2)

I see jobs are submitted to the cluster, however still getting this error. I checked the log (out.txt) file which shows the jobs and evaluation of doJobCollection functions i.e. :

Rscript -e 'batchtools::doJobCollection("/..pathtojobfile../joba7f53f7835b2d0e33a6e2fe563c6f6da.rds")'

I readRDS that file in R and its not empty , however execution of batchtools::doJobCollection("/..pathtojobfile../joba7f53f7835b2d0e33a6e2fe563c6f6da.rds") inside R is NULL. Do you have any idea if the problem lays in lsf.tmpl file or I am missing calling some other batchtools functions in my run.R script?