galyleo

Singularity containers

Singularity containers bring operating system-level virtualization to scientific and high-performance computing, allowing you to package complete software environments --- including operating systems, software applications, libraries, and data --- in a simple, portable, and reproducible way, which can then be executed and run almost anywhere.

If you have a Singularity container that you would like to run your Jupyter notebook session within, then you simply need to provide a path to the container with the --sif option in your launch command. This will start jupyter within the container using the singularity exec command. If necessary, you can also pass user-defined --bind mounts to the container and enable NVIDIA GPU support via the --nv flag.

One of the most powerful features of Singularity is its ability to convert an existing Docker container to a Singularity container. So, even if you are not familiar with how to build your own Singularity container, you can always search public container registries like Docker Hub for an existing container that may help you get your work done.

For example, let's say you need an R environment for your Jupyter notebook session. Why not try the latest r-notebook container from the Jupyter Docker Stacks project? To get started, you first use the singularity pull command to download and convert the Docker container to a Singularity container.

singularity pull docker://jupyter/r-notebook:latest

Once all of the layers of the Docker container have been downloaded and the container conversion process is complete, you can then launch your Jupyter notebook session with the newly built Singularity container.

galyleo launch --account abc123 --cpus 2 --memory 4 --time-limit 00:30:00 --sif r-notebook_latest.sif

On some systems like Expanse, you may need to load Singularity via the software module environment as well.

galyleo launch --account abc123 --cpus 2 --memory 4 --time-limit 00:30:00 --env-modules singularitypro --sif r-notebook_latest.sif --bind /expanse,/scratch

Here, the user-defined --bind mount option also enables access to both the /expanse network filesystems (e.g., /expanse/lustre) and the local NVMe /scratch disk(s) available on each compute node from within the container. By default, only your $HOME directory is accessible from within the container.

Singularity also provides native support for running containerized applications on NVIDIA GPUs. If you have a GPU-accelerated application you would like to run during your Jupyter notebook session, please make sure your container includes a CUDA-enabled version of the application that can utilize NVIDIA GPUs.

NVIDIA distributes a number of GPU-optimized containers via their container registry. This includes containers for all of the most popular deep learning frameworks --- PyTorch, TensorFlow, and MXNet --- with jupyter pre-installed. Like the the containers available from DockerHub, you can pull these containers to the HPC system you are working on

singularity pull docker://nvcr.io/nvidia/pytorch:21.07-py3

and then launch your Jupyter notebook session with galyleo. For example, you might want to run this PyTorch container on a single NVIDIA V100 GPU available in Expanse's gpu-shared partition.

galyleo launch --account abc123 --partition gpu-shared --cpus 10 --memory 93 --gpus 1 --time-limit 00:30:00 --env-modules singularitypro --sif pytorch_21.07-py3.sif --bind /expanse,/scratch --nv 

Note, however, how you request GPU resources with galyleo may be different from one HPC system to another. For example, on Comet you must use the --gres command-line option on Comet to specify both the type and number of GPUs required for your Jupyter notebook session. The following launch command would create a session within the NVIDIA PyTorch container on a single P100 GPU available in Comet's gpu-shared partition.

galyleo launch --account abc123 --partition gpu-shared --cpus 7 --gres gpu:p100:1 --time-limit 00:30:00 --sif pytorch_21.07-py3.sif --bind /oasis,/scratch --nv

In contrast, on TSCC you'll never explicitly request a specific number of GPUs for your Jupyter notebook session. All GPUs on TSCC are currently allocated implicitly in proportion to the number of CPU-cores requested by a job and available on the type of GPU-accelerated compute node you expect it to run on. And if you would like to request your notebook session be scheduled on a certain type of GPU, then you must pass the type of GPU required listed in the pbsnodes properties via the --constraint command-line option. For example, the following launch command will schedule your session on one of the NVIDIA GeForce RTX ] 2080Ti GPUs available in the gpu-hotel queue on TSCC.

galyleo launch --account abc123 --partition gpu-hotel --cpus 2 --constraint gpu2080ti --time-limit 00:30:00 --sif pytorch_21.07-py3.sif --bind /oasis --nv

Whatever you do, whenever you're launching your Jupyter notebook session with galyleo from a Singularity container on compute resources with NVIDIA GPUs, don't forget the include the --nv flag.

name: notebooks-sharing

channels:
  - conda-forge
  - anaconda

dependencies:
  - python=3.7
  - jupyterlab=3
  - pandas=1.2.4
  - matplotlib=3.4.2
  - seaborn=0.11.0
  - scikit-learn=0.23.2

galyleo launch --account abc123 --partition shared --cpus 4 --memory 8 --time-limit 00:30:00 --conda-env notebooks-sharing

galyleo launch --account abc123 --partition shared --cpus 4 --memory 8 --time-limit 00:30:00 --conda-env notebooks-sharing --conda-init miniconda3/etc/profile.d/conda.sh

galyleo launch --account abc123 --partition shared --cpus 4 --memory 8 --time-limit 00:30:00 --conda-env notebooks-sharing --conda-yml environment.yml

galyleo launch --account abc123 --partition shared --cpus 4 --memory 8 --time-limit 00:30:00 --conda-env notebooks-sharing --conda-yml environment.yml --mamba --cache