Changed good practice for shell hygiene

[wwarriner]

What is inaccurate?

Currently, the practice we have had good success with for shell hygiene on Cheaha is the following:

1. Clean shell, every time Have module reset in your .bashrc, and no other module load statements. This ensures that every script and terminal has a clean, well-defined, repeatable environment to start from.
2. Clean forks, every time After the preamble of every script using modules, have module reset, followed by other module load commands, followed by all other statements. This separates what software is loaded in the calling shell, from software loaded in the script shell. Be aware that forked processes (like scripts) and Slurm commands inherit the environment variables of the calling shell, including loaded modules.
3. Maintain default python executable Avoid module load Anaconda3 (or 2 for that matter) in .bashrc, don't use conda/mamba init, and don't put module load Anaconda3 nor conda/mamba activate in the "Environment Setup" box of OOD Interactive Apps. All of these change which python executable is used by default, which can result in hard-to-diagnose errors in OOD and in scripts.

Where is the inaccuracy?

This potentially benefits several pages. Find ways to share appropriate parts of the above on each page below.

1. https://docs.rc.uab.edu/cheaha/software/software/#using-anaconda
2. https://docs.rc.uab.edu/workflow_solutions/shell/
3. https://docs.rc.uab.edu/cheaha/software/modules/
4. https://docs.rc.uab.edu/cheaha/open_ondemand/ood_interactive/
5. https://docs.rc.uab.edu/cheaha/slurm/submitting_jobs/

There may be other pages.

[mdefende]

module reset shouldn't be included in .bashrc. This inhibits some debugging steps for shells that you actually want to inherit environments. For example, when testing whether modules load correctly in an OOD Rstudio app, you can use either a system command or open a built-in terminal in RStudio. However, both of these will run .bashrc AFTER inheriting the environment you want to check. The module reset in the .bashrc will reset the modules you're checking in this case.

I think its a better suggestion to have module reset at the beginning of Environment Setup windows (although there are multiple module reset commands in the script.sh.erb files for at least RStudio) or at the beginning of a batch script

[mdefende]

Suggestion 3 also would prevent RStudio from using reticulate which some packages need and is very helpful when needing user-installed so files from conda. module load Anaconda3 shouldn't be used with any Jupyter OOD jobs since it's already being loaded. Not sure about MATLAB, but it's plausible that the same situation of accessing an so file installed with conda might be useful there too. I think 3 should be amended to 'don't include module load Anaconda3 in Jupyter OOD jobs instead of all OOD jobs.

[wwarriner]

For post 1, fair point. I'll let folks know about that caveat when I discuss, and we can not propose the module reset in .bashrc. My thought was that this could help a lot of researchers who aren't ready for things like module debugging, to avoid common and hard-to-diagnose "dirty" environment issues.

I'll argue that debugging could happen in a separate environment. Personally, I do it all in a clean terminal. module reset, then all my module load then module -t list to check things. But I'll admit that R specifically is an odd case, because various modules interact directly with package management, so there can be more to debugging than just checking module versions and conflicts.

For post 2, that is 100% a fair point. I think we can limit my suggestion in (3) to only Jupyter notebooks. You're right about MATLAB. MATLAB has some serious Python integration these days (I guess they've been watching Microsoft). I'm fairly sure it does not really "know" about Anaconda, and there isn't a way to "activate" an environment within the MATLAB context. It's possible to use system() to do it, but that would be within a forked process, so no impact on the parent MATLAB session. What folks can do is use pyenv to discover their Anaconda environments and activate them that way. Though the pythonpath variable may need to be set manually to find installed packages. I'm going to test briefly and get back on this, cause I'm curious.

[wwarriner]

Using module load Anaconda3 I see the following in MATLAB. The path for the executable is expected. It points to the base environment.

>> pyenv

ans = 

  PythonEnvironment with properties:

          Version: "3.9"
       Executable: "/share/apps/rc/software/Anaconda3/2022.05/bin/python"
          Library: "/share/apps/rc/software/Anaconda3/2022.05/lib/libpython3.9.so"
             Home: "/share/apps/rc/software/Anaconda3/2022.05"
           Status: NotLoaded
    ExecutionMode: InProcess

Using system("source activate <env>") has no effect on pyenv. On a side note, it is strange I had to use source activate. The usual conda activate threw the "use conda init" error. I suppose because the shell process created by system() isn't a tty shell, but a process forked based on the first part of the input string, so things like .bashrc aren't being used. I tested with system("ps -p $$") and got back ps as the process. Typically it would give back bash.

If I also add conda activate <env> to the env setup box, then I get the following. We can see it points to the environment I activated.

>> pyenv

ans = 

  PythonEnvironment with properties:

          Version: "3.7"
       Executable: "/home/wwarr/.conda/envs/py37/bin/python"
          Library: "/home/wwarr/.conda/envs/py37/lib/libpython3.7m.so"
             Home: "/home/wwarr/.conda/envs/py37"
           Status: NotLoaded
    ExecutionMode: InProcess