r-quadros
commented
1 year ago Modify git version in docker image, convert to singularity and run in cluster
- Get the docker image first, pull a suitable image with tag (example below)
docker pull mdolab/public:u22-gcc-ompi-stable
-
Create a container and change the git version in there for the Pyoptsparse Run the pip install command to install the earlier version of pyoptsparse library
-
Now create a new image from the modified container
docker commit container-ID image-name
Details:https://www.skynats.com/blog/edit-docker-image/
-
Tar the docker image, transfer to hpc and create a singularity file Details:https://stackoverflow.com/questions/52738888/converting-docker-tar-file-into-a-singularity-image
-
Use this singularity file in qsub script. Key line from the qsub script (note the $BASHRC_MDOLAB that is needed when running singularity file):
singularity exec --writable-tmpfs $HOME/Images/SingularityV3/MachAero.sif /bin/bash -c 'source $BASHRC_MDOLAB; mpirun -np ${cpus} python $HOME/MACHAeroV3/Opt/aero_opt.py' > output.dat
Description of feature
The cluster at my university encourages the use of singularity which is an open source container platform used by many HPC facilities. I have converted the docker image into a singularity file using the command: singularity pull docker://mdolab/public:u22-gcc-ompi-stable
However, when I load the image into a container using the command: singularity shell public_u22-gcc-ompi-stable.sif
I see that I have no access to the basic libraries such as "mpirun" which the docker container clearly shows. Is there any other way of converting the docker image into a singularity *.sif file that you would suggest?
If a singularity file is tested and provided that would be immensely helpful to those who would want to take MACHAero to the cluster.
Many thanks, Russell