Create a custom conda environment on the cluster

[samwaseda]

This tutorial details how to create a custom conda environment on the cluster and use it for calculations:

• Use the directory /cmmc/ptmp/<username> where <username> is your username. This is to make sure you do not exceed the file limits.
• Go to the directory cd /cmmc/ptmp/<username>
• Create an environment with command conda create -p ./<envname> (specify python version if needed)
• activate env with source activate <envname>
• To speed up things greatly, install mamba first, conda install -c conda-forge mamba
• install ipykernel for later, mamba install -c conda-forge ipykernel
• install other packages as needed
• install the environment as a kernel: python -m ipykernel install --user --name <envname>
• Go to /u/<username>/.local/share/jupyter/kernels/<envname>
• Open the file kernel.json and add:

  "env": {
  "PATH": "/cmmc/ptmp/<username>/<envname>/bin:${PATH}",
  "CONDA_PREFIX": "/cmmc/ptmp/<username>/<envname>"
  }

  make sure to enclose it within the global {}

• Now go to jupyterhub, File -> hub control panel -> Stop server -> start again
• Open a new notebook, choose the kernel <envname> and have fun!

[jan-janssen]

The NERSC computing center also has a nice how-to on this topic https://docs.nersc.gov/services/jupyter/how-to-guides/#how-to-use-a-container-to-run-a-jupyter-kernel

[jan-janssen]

Be careful, if the kernel.json file is wrongly formatted, no kernels are displayed in the Jupyter lab environment. Here is an example of a correctly formatted kernel.json file:

{
 "argv": [
  "/cmmc/ptmp/janj/mambaforge/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "janj",
 "language": "python",
 "metadata": {
  "debugger": true
 },
 "env": {
  "PATH": "/cmmc/ptmp/janj/mambaforge/bin:${PATH}",
  "CONDA_PREFIX": "/cmmc/ptmp/janj/mambaforge"
 }
}

[srmnitc]

Make sure to not have module load pyiron/dev in your bashrc. Instead do, module load pyiron/latest.

[srmnitc]

In case of errors such as:

InvalidArchiveError("Error with archive /cmmc/u/<username>/.conda/pkgs/python-3.11.8-hab00c5b_0_cpython/.cph_tmps783dxu6/pkg-python-3.11.8-hab00c5b_0_cpython.tar.zst.  You probably need to delete and re-download or re-create this file.  Message from libarchive was:\n\nCan't create 'lib/python3.11/lib2to3/fixes/__pycache__/fix_ne.cpython-311.pyc'")

Run conda clean --all and run the steps again.