[meta] Carefully consider how to handle environments in non-user-privileged locations (HPC etc)

[gdevenyi]

As a sysadmin trying to use conda to "package" various standardized environments, and take advantage of environment-modules/lmod to enbale/disable them, recent changes to conda, requiring deep hooks into shells have made my job much more difficult. In addition, users running conda to create their own environments can run into permissions issues when the conda cache directories don't have write permissions.

I'm opening this bug as a meta-location to discuss the design choices and adjustments that might be made to best enable these kinds of use-cases.

[wolfv]

Hi @gdevenyi thanks for starting this discussion.

I think you have there are two points you're making:

• handling of non-writable package cache: we've in fact improved on this point recently and micromamba & mamba now support multiple package caches very well – we're looking for the first writable cache and write there if necessary, otherwise we just read from the existing cache
• and yes, some packages need specific environment variables set to given values – which is why they run activation scripts when the environment is activated. Do you have ideas on how to improve this? I started the multisheller project some time ago to have a higher level way to write these environment modifications for multiple shell languages, but I am not sure that would help in your case :) You can find all activation scripts in a certain location ($CONDA_PREFIX/etc/conda/activate.d/ ).

[adriendelsalle]

See Handle multiple package cache and envs dirs #1110

[gdevenyi]

and yes, some packages need specific environment variables set to given values

Yes, there's that, but in addition, I used to be able to just enable a conda env with conda activate with conda on PATH. Now conda must have its hooks in the bash env, which makes things more difficult.

Glad to see these ideas are being considered :+1:

[wolfv]

@gdevenyi you can still do that but you will miss activation scripts. That can have bad consequences for some packages ... but many packages will also just work fine.

[davidrpugh]

Working at HPC center where Conda/mamba is the primary mechanism for users to install software. Users install Conda/mamba in their home directories (including initializing Conda for bash), these home directories are available on the compute nodes, and then users launch jobs via Slurm. The "trick" that we use to make conda activate command work as expected is to launch all jobs inside a login shell so that .bash_profile and .bashrc are properly sourced.

#!/bin/bash --login
# SBATCH ...

conda activate $ENV_PREFIX

Hope fully this little trick helps others!

[nlvw]

@davidrpugh For our HPC center we install miniforge centrally and use lmod software modules to load it into the user environment. Basically we source the profile.d activation script on load then we manually unset/unalias at unload. This is specific to conda and miniforge so you would need to modify it for mamba/micromamba.

help([[For help contact hpc-team@nmsu.edu]])
whatis([[Name : Miniforge]])
whatis([[Version: 3-4.11.0-4]])
whatis([[Target: x86_64]])
whatis([[Short description : Miniforge is a free and open-source distribution of the Python programming language for scientific computing (data science, machine learning applications, large-scale data processing, predictive analytics, etc.), that aims to simplify package management and deployment. Package versions are managed by the package management system conda.]])

family("python")
conflict("anaconda","anaconda2","anaconda3","miniconda","miniconda2","miniconda3","python","python2","python3","miniforge","miniforge2","miniforge3")

local base = "/software/anaconda/miniforge/3-4.11.0-4"
local myShell = myShellName()

setenv("ANACONDA_ROOT",base)
prepend_path("PATH",pathJoin(base,"bin"))

-- this happens at load
if (conda_exe == nil or conda_exe == "") then
    if (myShell == "bash") then
        cmd = "source " .. base .. "/etc/profile.d/conda.sh"
    else
        cmd = "source " .. base .. "/etc/profile.d/conda.csh"
    end
    execute{cmd=cmd, modeA={"load"}}
end

-- this happens at unload
if (myShell == "bash") then
    if (mode() == "unload") then
        remove_path("PATH", pathJoin(base,"condabin"))
        remove_path("PATH", pathJoin(base,"bin"))
    end
    cmd = "conda deactivate; " ..
          "unset $(compgen -v | grep -i '^CONDA_'); " ..
          "unset $(compgen -v | grep -i '^_CONDA'); " ..
          "unset $(compgen -v | grep -i '^__CONDA'); " ..
          "unset _CE_CONDA; unset _CE_M; unset -f conda"
    execute{cmd=cmd, modeA={"unload"}}
else
    cmd = "conda deactivate; " ..
          "unsetenv CONDA_EXE; unsetenv _CONDA_ROOT; unsetenv _CONDA_EXE; " ..
          "unsetenv CONDA_PYTHON_EXE; unset CONDA_SHLVL; unalias conda"
    execute{cmd=cmd, modeA={"unload"}}
end

@wolfv It would help micromamba generated a deactivation script that undid any changes that the activation (etc/profile.d/micromamba.sh) does to the user environment.

[jonashaag]

To the future readers: There are at least 50 other relevant other tickets to be found. Feel free to post some here :)