mschubert / clustermq

R package to send function calls as jobs on LSF, SGE, Slurm, PBS/Torque, or each via SSH
https://mschubert.github.io/clustermq/
Apache License 2.0
146 stars 27 forks source link

How to enable socket authentication with singularity containers #226

Closed mattwarkentin closed 3 years ago

mattwarkentin commented 3 years ago

Hi @mschubert,

Where should I add CMQ_AUTH={{ auth }} in my Slurm template when running the workers inside a singularity container to avoid the warning shown below? I know it typically comes before the R command, but this caused issues with singularity.

Warning in private$fill_options(n_jobs = n_jobs, ...) :
  Add 'CMQ_AUTH={{ auth }}' to template to enable socket authentication

Here is my current template:

#!/usr/bin/bash

#SBATCH --job-name={{ job_name }}
#SBATCH --partition={{ partition }}
#SBATCH --nodelist={{ node }}
#SBATCH --time={{ time | 0 }}
#SBATCH --array=1-{{ n_jobs }}
#SBATCH --output={{ log_file | /dev/null }}
#SBATCH --error={{ log_file | /dev/null }}
#SBATCH --mem-per-cpu={{ memory | 4096 }}
#SBATCH --cpus-per-task={{ cores | 1 }}
#SBATCH --export=ALL

module load singularity

ulimit -v $(( 1024 * {{ memory | 4096 }} ))
singularity exec cv-na.sif R --no-save --no-restore -e 'clustermq:::worker("{{ master }}")'
mschubert commented 3 years ago

Does

export CMQ_AUTH={{ auth }}

on a separate line before the singularity call work?

mattwarkentin commented 3 years ago

Yes, that worked! Thanks.