Docker image on Singularity

[FrancescoCasalegno]

Context

We are now able (see #47, #49) to create a docker image and run it in a container on our workstations. However, we also need to be able to run the image on BB5, where instead of docker we have singularity[^1].

Actions

The following are the instructions to follow (in case of issues, please contact @GianlucaFicarelli).

1. (Apparently not needed any more) ~Ask Core Services to add you and the other interested users to the list of allowed users for singularity.~
2. Clone this repo on GitLab (under https://bbpgitlab.epfl.ch/ml). There are two ways to achieve this.
   1. (Easy way) Clone the GitHub repo locally, then push it to a brand new repo on Gitlab (or create a fork from our GitHub in GitLab). Easy and quick, but at some point the GitHub and GitLab repos will diverge.
   2. (Hard way) Create a GitLab mirror of our GitHub repo. May be a bit harder, but will not diverge from GitHub.
3. Go to the GitLab repo, on the sidebar select "Packages & Registries" > "Container Registry".
4. Follow the instructions on the GitLab page of "Container Registry" to docker push a locally built Docker image (it will be something like docker push bbpgitlab.epfl.ch:5050/ml/Deep-Atlas).
5. Connect via ssh to BB5, allocate a node with salloc, and load the module for singularity:

   module load unstable singularityce

6. Run the singularity image in a container[^2][^3]

   singularity pull --docker-login --no-https docker://bbpgitlab.epfl.ch:5050/<group>/<project>:<tag>
   singularity exec --cleanenv --containall --bind $TMPDIR:/tmp,/gpfs/bbp.cscs.ch/project <image>.sif bash

7. Try to run the pipeline and check that everything works fine.

[^1]: Recently singularity became part of apptainer, but the fork for singularity is still maintained as separate from the apptainer fork. Documentation and interfaces for the two forks are mostly interchangeable. Anyway, at BBP we apparently prefer to stick with the singularity fork. [^2]: By default, singularity passes all environment variables to the environment within your container. To avoid this, we use --cleanenv. [^3]: The --containall option allows you to run the software in the container with only minimal access to the host operating system. One of the filesystems which are no longer mounted within the container is /tmp, but we can still bind it by adding --bind $TMPDIR:/tmp.

[drsantos89]

1) The Deep-Atlas repo is now cloned on GitLab as a mirror from the GitHub repo. We followed instructions here and here. The token has a validity of 90 days and will need to be renewed.

2) For Deep-Atlas developers: 2.1) The Deep-Atlas docker container can be found at "Packages & Registries" > "Container Registry" 2.2) To update the docker image, run the following commands (on the local machine): 2.3) docker login bbpgitlab.epfl.ch:5050 2.4) docker build -f docker/Dockerfile -t bbpgitlab.epfl.ch:5050/ml/deep-atlas 2.5) docker push bbpgitlab.epfl.ch:5050/ml/deep-atlas

3) For Deep-Atlas users (on BB5): 3.1) Connect via ssh to BB5, allocate a node with salloc, and load the module for singularity: module load unstable singularityce. 3.2) Pull the docker image from GitLab and convert it to a singularity image: singularity pull --docker-login --no-https docker://bbpgitlab.epfl.ch:5050/ml/deep-atlas:latest 3.3) Run the singularity container: singularity exec --cleanenv --containall --bind $TMPDIR:/tmp,/gpfs/bbp.cscs.ch/project,$HOME:/home deep-atlas_latest.sif bash 3.4) There is no write access inside the singularity container, so results have to be written, for instance, to the /home directory or inside of projXX using the mounted gpfs filesystem (see option --bind above).

@GianlucaFicarelli @mgeplf

[FrancescoCasalegno]

Hi @EmilieDel @drsantos89 ,

Before closing this issue I would suggest the following steps.

• [x] Update Dockerfile by replacing MY_USER_ID with DEAL_USER_IDS, and let people know (add info relative to it in README.md) that this is a comma-separate list of <username>/<userid> such as "$(whoami)/$(id -u)".
• [x] In the Dockerfile, we can have a line testing for the variable to be define and print a warning like so:

  RUN test -n "$DEAL_USER_IDS" || echo "WARNING: variable DEAL_USER_IDS was not set, creating only default user 'guest/1000' !"

• [ ] Update README.md with info on how to use Singularity as a user on BB5 (i.e. more or less the content of point 3. in this comment) in a sub-section of the "Docker" section.
• [x] Make sure that we have write access to mounted volumes, namely the home and projXX folders on gpfs.
• [ ] Try salloc a node with GPU (I think it should be -C volta but double check that on Confluence). Can we then run with the GPU in the Singularity container? Are the nvidia libraries all correctly set up?

[EmilieDel]

I confirm that we can write on /gpfs/ in Singularity container.

[drsantos89]

GPU working with --nv flag on singularity tested with: salloc -N 1 -A proj101 -p interactive -C volta -t 1:00:00 --ntasks-per-node 36 singularity exec --cleanenv --nv --containall --bind $TMPDIR:/tmp,/gpfs/bbp.cscs.ch/project,$HOME:/home deep-atlas_latest.sif bash (docker base image was also updated from cuda 10.2 to 11.4, to match bb5)

[FrancescoCasalegno]

(docker base image was also updated from cuda 10.2 to 11.4, to match bb5)

Also tried with cuda 10.2, and even if doesn't match the CUDA version of the underlying system it seems to work perfectly fine.