simplify workflow for singularity users?

[biocyberman]

We have a ComputeCluster with Singularity support. It is obvious that we don't want to give root access (sudo) to everyone. However, it is a hassle for users to do this:

1. Install singularity on their own machines (Windows, MacOS, Linux)
2. Create a git repo or a directory on user's PC.
3. Write/Edit Signularity definition file
4. Build with root/sudo something like this sudo singularity build imagename.sif Singularity.def
5. Upload imagename.sif to ComputeCluster.
6. Test magename.sif on ComputeCluster
7. Repeat 3,4,5,6 until everythin works.

What I am trying to setup is like this:

1. Create a git repo or a directory on ComputeCluster or local PC.
2. Write/Edit Signularity definition file.
3. Commit and push to our local (gitea) server.
4. (Automagically) Build and push imagename.sif to our local sregistry.
5. Pull and Test magename.sif on ComputeCluster
6. Repeat 2,3,5 until everythin works.

So, even though both two approaches have almost the same number of steps. But the second approaches requires much less work for ComputeCluster users and no root required.

The question is, how do I setup the magic at step 4 (second approach)? I've checked that the only Continuous Integration solution that Gitea seems to support at the moment is Drone. But sregistry doesn't seem to support Drone yet. So I would like to hear comments and suggestions.

FYI, actually before getting sregistry up and running, I thought it would offer automated build like what Docker Hub offer: https://docs.docker.com/docker-hub/builds/

[vsoch]

hey @biocyberman - I'd be happy to discuss this with you! The "automated build like Docker" is something that Singularity Hub offers, but I'd more strongly recommend that we try to figure this out with Singularity Registry Server, as it's an open source community code base (and Shub is not). Using Gitea and Drone, the easiest method would be to set up CI to test and build the container (there are many examples here) and the user would just be required to do something like add an encrypted environment variable to their build context associated with each repo (for some CI this would be as simple as setting the build context once, and copy pasting the config files to do the build). The more complicated thing would be to have a commit hit some webhook to do the build (akin to Docker Hub / Singularity Hub), but you still have the problem of where the build actually happens (CI would be the perfect environment to do it). So here is the pipeline that I'm proposing, and I can help you to make recipe for your CI (although I have never used it before!)

0. (once) the user adds their token to be an encrypted environment variable in gitea, available to all repos with the context enabled.
1. user creates repository with config file that will build (optionally test) and then push on success.
2. Any time either a commit is done to master, OR a pull request that is reviewed and merged, the image is built and pushed
3. The user then pulls to the cluster.

Singularity Registry Server is open source and community focused, so most of the features there were asked for by the community, and the same could be true for any others that are needed! I/we develop it based on what users ask for and need. Let me know your thoughts on the above!

[biocyberman]

Hi @vsoch Great to know you are willing to figure this out. Here are what I've checked so far:

1. Installed a Drone instance and connected to our Gitea server. It was easy enough.
2. Check other discussions regarding Drone + Singularity. Several issues have been submitted regarding this and still no viable solution yet.
3. Check support for Singularity support from Drone's side: https://discourse.drone.io/t/drone-io-and-singularity/1549 It doesn't seem to be trivial to add support for Singularity runtime.

Therefore, things we need to find out:

1. Can we use Drone to build Singularity images with setup.sh and build.sh like in other CI examples. If this can be done, I would like an example for this. I can also help to work this out.
2. Can we set some effort to speed up having official Singularity support from Drone?

For the time being, I also testing a setup where normal user can only run sudo with sudo /usr/local/bin/singularity build * command. What is the security consequence of this? Can a user pack and unpack malicious programs during build (e.g. a root-escalating shell)? I am still in doubt and trying to find out. What is your comment about this?

[vsoch]

The article you link is interested in using Singularity as a container base for the build itself, which isn't what we want - we would be good to use Docker (with privileged) or a machine to build a Singularity container on top of it. I think we should set up a test - are you able to give me the ability to create an account on your Gitea server that is connected to drone so I can write and test a recipe? Our goal would be to have a recipe that fits alongside the ones that you linked in the Singularity CI repository.

The build working or not will come down to the permissions that Drone allows for it's containers / servers. If you are able to use sudo, that's a good start (and sudo is required for builds with recipes, so if there are any security issues it's on Drone to ensure the build is isolated, etc.) I am not a security expert so I don't have further comment, but I'll say that generally building with sudo isn't an issue on other CI platforms.

Let me know about the above and we will start testing!

[biocyberman]

Great! @vsoch I will do some setup and coordinate with my colleagues before getting back to you on email about server setup. Are you active on this email address: ? Please expect some emails from vle @ its. aau. dk by the of next week.

[vsoch]

Yes that’s me! :)

[vsoch]

hey @biocyberman I'm going to close the issue here, if you have further questions / help you know how to reach me :) Note that I've redacted my email in the above just so it's not that public.