Allow simultaneous use of envmodules and conda environment

[gatoniel]

Is your feature request related to a problem? Please describe. On our cluster, certain libraries are pre-installed and can be loaded via module load while python environments are managed via conda. In my specific use case, I rely on loading CUDA and cuDNN via module load and managing installation of tensorflow and other python modules via conda environments. This has worked in previous snakemake versions by providing custom submit.py scripts in the profiles. Now I wonder if I can simultaneously use the envmodules and conda directives in rules.

The docs sound as if these two directives are mutualy exclusive. Is there a way for me, to use snakemake v8 and this executor plugin for the above mentioned requirement?

Describe the solution you'd like Execute a job and simultaneously provide cluster modules and a conda environment.

Describe alternatives you've considered I would go back to earlier snakemake versions where I used custom submit.py scripts.

Additional context Furthermore, I want to ask whether the executor runs module purge before loading the modules as is recommended on many clusters. I couldn't find the info in the docs.

Best, Niklas

EDIT: I know of this quote:

Using environment modules can be combined with conda and apptainer (--sdm env-modules conda apptainer), which will then be only used as a fallback for rules not defining environment modules. For running jobs, the squeue command: from https://snakemake.github.io/snakemake-plugin-catalog/plugins/executor/slurm.html

But I wonder if this really allows for simultaneous usage of envmodules and conda in the same rule, or whether one is ussed only as fallback of the other?

[gatoniel]

Ok, tested it, and it seems that with --sdm env-modules conda and with --use-conda --use-envmodules modules and conda environment are loaded simultaneously in the same rule.

Leaves me with the question, wheter module purge is run in the very beginning.

[cmeesters]

yes, modules are purged prior to loading, albeit not by this executor, but by Snakemake itself and this is not an option.

I will have a look at this, when I return (being on a mini-holiday, now). The idea is not exactly, that when specifying both options, that they are mutually exclusive, but rather, that --sdm env-modules,conds attempts to load a module and falls back to conda, if there is no env-module specified (that, at least, was my understanding).

[cmeesters]

There is no update on the issue — yet. We are currently refactoring the software support code in general.