Getting started with plan("batchtools_lsf")

[fboehm]

I'd like to use futures on a LSF computing cluster. I see that one may use

plan("batchtools_lsf")

ahead of functions that use futures.

I'm struggling to identify the needed directory path for the file lsf.tmpl. I'm also struggling to understand the role of the template file.

Right now, my workflow involves typing

bsub < mysubmit.bsub 

at the shell prompt to submit jobs.

The text in mysubmit.bsub is:

#BSUB -q short 
#BSUB -o %J.out 
#BSUB -e %J.err 
#BSUB -N
#BSUB -W 1:00
#BSUB -n 2
#BSUB -R rusage[mem=8000]
#BSUB -R span[hosts=1]

module load gcc/8.1.0
module load libpng/1.6.8
module load pandoc/2.7.2
module load R/4.0.0_gcc
module load cairo/1.12.16
Rscript -e 'rmarkdown::render("../Rmd/bivariate-mapping-greenyellow-magenta-Chr10.Rmd")'

where I use R futures in the ../Rmd/bivariate-mapping-greenyellow-magenta-Chr10.Rmd file. I also call plan("batchtools_lsf") in advance of using futures in the Rmd's R code.

I get an R error message; here is the full output in the .err file:

gcc 8.1.0 is located under /share/pkg/gcc/8.1.0
pandoc 2.7.2 is located under /share/pkg/pandoc/2.7.2
R 4.0.0_gcc is located under /share/pkg/R/4.0.0_gcc
When compiling modules for this, be sure to load gcc/8.1.0
cairo 1.12.16 is located under /share/pkg/cairo/1.12.16

processing file: bivariate-mapping-greenyellow-magenta-Chr10.Rmd
Quitting from lines 54-61 (bivariate-mapping-greenyellow-magenta-Chr10.Rmd)
Error in batchtools_by_template(expr, envir = envir, substitute = FALSE,  :
  Failed to locate a batchtools template file: *lsf.tmpl
Calls: <Anonymous> ... future_map_template -> future -> makeFuture -> batchtools_by_template
In addition: Warning message:
In grDevices::png(f) : unable to open connection to X11 display ''
Execution halted

Apparently, the *lsf.tmpl file is not found.

I currently have a text file, lsf.tmpl, without a preceding period, in the same directory as mysubmit.bsub. lsf.tmpl contains the text:

## Default resources can be set in your .batchtools.conf.R by defining the variable
## 'default.resources' as a named list.

## https://raw.githubusercontent.com/mllg/batchtools/master/inst/templates/lsf-simple.tmpl

#BSUB -J <%= job.name %>                             # Name of the job
#BSUB -o <%= log.file %>                             # Output is sent to logfile, stdout + stderr by default
#BSUB -q <%= resources$queue %>                      # Job queue
#BSUB -W <%= round(resources$walltime / 60, 1) %>    # Walltime (LSF requires minutes, batchtools uses seconds)
#BSUB -M <%= resources$memory %>                     # Memory requirements, e.g. "5000KB", "500MB", "5GB" etc. 

## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>

<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>

Rscript -e 'batchtools::doJobCollection("<%= uri %>")'

I copied lsf.tmpl verbatim from https://github.com/mllg/batchtools/blob/master/inst/templates/lsf-simple.tmpl

I'm not using a configuration file currently.

Should my workflow still begin with me submitting, via bsub, mysubmit.bsub? Or do I need to alter the workflow when using plan("batchtools_lsf")?

Is it correct to use, without modifications, the current version of lsf.tmpl? Or do I need to somehow tailor it to my uses? I'm not sure exactly what role the template file plays.

I appreciate any suggestions that you offer. Thank you!

[cfljam]

Hi there, On our LSF cluster this works with the default template file batchtools.openlava.tmpl in the same directory

$ module unload R; module load R/3.6.1;Rscript -e 'library(future.batchtools);plan(batchtools_openlava);plan(batchtools_openlava); x %<-% { Sys.sleep(5); 3.14 }; y %<-% { Sys.sleep(5); 100 };x+y'
Loading required package: future
[1] 103.14

[fboehm]

Thank you, @cfljam! I recently changed jobs, so I no longer have access to an LSF cluster, so I can't try it right now. Regardless, I really appreciate your reply!