Build singularity containers for each model

[simonrolph]

So looking at the documentation, singularity build is not available on LUMI so we have to build it ourselves on our own hardware then transfer to LUMI. https://docs.lumi-supercomputer.eu/software/containers/singularity/ singularity can only be installed on linux https://docs.sylabs.io/guides/3.0/user-guide/installation.html so maybe we could install it on datalabs? Or get someone with LUMI knowhow to build it for us.

Docker files to build singularity containers from:

• [x] Biodiversity model: https://github.com/BioDT/uc-ces/blob/main/biodiversity_model/Dockerfile
• [x] Recreation model: tbc

[simonrolph]

Notes copied from an email from @trossi

For converting docker container to singularity, two alternative approaches below could be used. I'd recommend the second approach as it's more aligned with the BioDT architecture plan (although the "final" container repository might be something else than github, the approach is still the same). For testing either way should work. I added an example version number 0.1.0 to commands below to keep track of the container as it evolves

1. Create singularity container image file (sif) on a local machine and transfer it to LUMI:

# Create ces-biodiversity_0.1.0.sif
docker build -t ces-biodiversity:0.1.0 .
docker save ces-biodiversity:0.1.0 -o temp.tar
singularity build ces-biodiversity_0.1.0.sif docker-archive://temp.tar

# Copy .sif file to LUMI
scp ces-biodiversity_0.1.0.sif lumi:/projappl/project_XXXXXXXX/_my_directory_/

2. Push docker image to a repository and pull it to LUMI:

   
   # Example for pushing to BioDT github

Build with correct tag and labels

docker build \ --label "org.opencontainers.image.source=https://github.com/BioDT/uc-ces" \ --label "org.opencontainers.image.description=BioDT Cultural Ecosystem Services pDT - Biodiversity Component" \ -t ghcr.io/biodt/ces-biodiversity:0.1.0 .

Push (login requires github access token with scope 'write:packages', see

https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token#creating-a-personal-access-token-classic

docker login ghcr.io docker push ghcr.io/biodt/ces-biodiversity:0.1.0

Then container should show up in https://github.com/BioDT/uc-ces in "Packages" panel

Pull image on LUMI (login is with github access token again)

singularity pull --docker-login docker://ghcr.io/biodt/ces-biodiversity:0.1.0

File ces-biodiversity_0.1.0.sif should have been generated

[simonrolph]

Example run using singulairty:

singularity exec --bind "$PWD" /projappl/project_465000357/simonrolph/ces-biodiversity_0.1.0.sif Rscript run_biodiversity_model.R 5334220 outputs/maps outputs/reports 5

Then set up as a slurm script: https://docs.lumi-supercomputer.eu/runjobs/scheduled-jobs/slurm-quickstart/

No modules as we're using a container

[WillOnGit]

The docker container I've built works fine on my machine but doesn't seem to work on LUMI for some reason.

[trossi]

The docker container I've built works fine on my machine but doesn't seem to work on LUMI for some reason.

What is the error message?

[WillOnGit]

I figured it out, it's because of the way the Docker image was relying on WORKDIR which is different with Singularity. Adding the --pwd flag has fixed it! Now just to get it running via SLURM.

[WillOnGit]

All running successfully