singularity support in the glidein

[dsschult]

Singularity would allow us to convert between OS types. There are hooks in condor 8.6 to support jobs requesting singularity. Figure out how to make this work.

[jvansanten]

This how-to provides a fairly good overview. Note that Singularity does not like the automounter, so if you eval $(/cvmfs/icecube.opensciencegrid.org/py2-v2/setup.sh) inside the container on a system that uses autofs to mount CVFMS repos, you'll get

sh: /cvmfs/icecube.opensciencegrid.org/py2-v2/setup.sh: Too many levels of symbolic links

A bad workaround is to cat the setup script on the host immediately before entering the container to force the repo to mount.

[dsschult]

We probably need to only enable this on appropriate kernels anyway, so I'd just not enable Singularity on systems that don't support automouting correctly (and use automounting).

Note that if it's already mounted, that should be fine. Since part of the glidein setup is to check for cvmfs, this might work fine for us.

[jvansanten]

Does the slot have to be able to run Singularity containers as jobs, or is it enough to run the glidein inside a container to paper over a weird site OS? In the latter case, it should just be a fairly simple change to glidein_start.sh, but that would also preclude power users from running their own containers.

[dsschult]

We probably want both?

Running the entire glidein inside singularity allows us to get around strange issues of the local OS.

Running the individual jobs in singularity is more optimal for the user, since they can specify the OS they want at late binding.

[briedel]

Another thing to add here is that we might want to add a custom classad for people to run their own containers

[dsschult]

Note that running singularity inside singularity doesn't work with kernels that don't support user namespaces. This is sadly most computers out there, since they've default disabled it on newer kernels because of security fears.

If we're forced to start the glidein inside a container, we might not be able to support custom user containers.

[briedel]

I guess you mean OSG having singularity inside singularity?

[dsschult]

Or the example of Cori, or Cedar, where we need to run a centos7 singularity image just to start the glidein. Then you can't let the user job specify an additional container.

[dsschult]

Running the entire glidein inside singularity now works. Thanks to @jamierajewski