search

mllg / batchtools

Tools for computation on batch systems
https://mllg.github.io/batchtools/
GNU Lesser General Public License v3.0
169 stars 51 forks source link

Creating jobs without package dependencies installed on head node #298

Open lpiep opened 6 months ago

lpiep commented 6 months ago

I have a use-case where I'd like to be able to create and submit jobs from the head node of a cluster that does not itself have the packages being used by the compute nodes. In this case, it's because the dependencies are a PITA to set up, and so I have built a singularity container for running them. Per https://github.com/HenrikBengtsson/future.batchtools/issues/44, I've put the singularity command in my template file and can submit jobs to be run in the container with batchMap and submitJobs.

However, I run into a problem when defining jobs with batchMap using functions not installed on the head node.

Assume pita_package contains pita_function and can't be installed on the head node. Then a call to batchMap results in 

> batchMap(pita_package::pita_function, 1:10)
Error in loadNamespace(x) : there is no package called ‘pita_package’

I've been able to work around this by creating a wrapper function for pita_function. E.g. 

> pita_wrapper <- function(...) pita_package::pita_function(...) 
> batchMap(pita_wrapper, 1:10)
Adding 10 jobs ...

I'm adding this issue in part to help anyone runs into this question in the future, and in part to see if it makes sense to implement some sort of lazy evaluation into batchMap and its cousins to avoid the workaround.