search

mschubert / clustermq

R package to send function calls as jobs on LSF, SGE, Slurm, PBS/Torque, or each via SSH
https://mschubert.github.io/clustermq/
Apache License 2.0
146 stars 27 forks source link

How to modify slurm template to run inside singularity container #214

Closed mattwarkentin closed 3 years ago

mattwarkentin commented 3 years ago

Hi @mschubert,

I was just wondering how I should modify my slurm template file in order to run the R worker processes inside a singularity container. Does the below look correct? Basically just taking the entire last line in the default template file and executing it with singularity exec container.sif ....

#!/usr/bin/bash

#SBATCH ...
#SBATCH ...

module load singularity

ulimit -v $(( 1024 * {{ memory | 4096 }} ))
singularity exec container.sif CMQ_AUTH={{ auth }} R --no-save --no-restore -e 'clustermq:::worker("{{ master }}")'
mschubert commented 3 years ago

That's how I think it should work!

Note that the container needs to be able to access the network on the host system.

mattwarkentin commented 3 years ago

Okay, thanks! I'm about to test it out so I will report back with how it goes.

mattwarkentin commented 3 years ago

Okay so it works well, but only after I removed the CMQ_AUTH={{ auth }} from the final line. Otherwise it produces this error:

/.singularity.d/actions/exec: 9: exec: CMQ_AUTH=ztbsf: not found