SingularityRequirement

[vinjana]

We would like to run containers safely in a multi-user LSF cluster. Docker has many security issues stemming from dockerd being run as root. Singularity is an alternative that gains more and more popularity in Science.

In the foreseeable future we need a way to execute Singularity containers (and thus by Singularities features also Docker!) in our multi-user LSF cluster. Thus, we need a SingularityRequirement analogous to the DockerRequirement in e.g. cwtool.

[adamsla]

Support is already in LSF 10.1.0.x, I would make x 6+. You simply setup your container specification in the LSF Application profile, and then in your cwl wrapper, you include that application profile with LSF's extended attributes around the cwl piece. We have addressed all the security issues in Docker inside of our LSF integration with it.

I hope that helps.

[skeeey]

Currently CWL specification does not have the SingularityRequirement, as an implementation, cwltool provides a command flag (--singularity) to use Singularity runtime for running containers, this still use DockerRequirement to define the container runtime.

For cwlexec, we may also provide a flag to specify the container runtime

[vinjana]

@skeeey About the actual implementation I do not care so much, as long as it allows to run Singularity containers on LSF with little effort and adaptation to our workflows :-). My impression is that support for Singularity is not really part of the CWL standard. Therefore, if the --singularity + DockerRequirement option fits into the usage concept of cwlexec, this solution would be perfectly fine.

@adamsla Thanks for the hint concerning the profiles! We don't have much experience with application profiles yet, but will certainly investigate in that direction. Concerning your statement about Docker security, however, I am a bit confused. Your statement contradicts what we were told by a pen-tester and security consultant (although on the Docker topic in general). In principle these issues are with the dockerd and that it runs with root privileges, right? Would you hint me at a document or description explaining in detail, how LSF tackles the issue?

[mr-c]

Hello @vinjana, I'm the CWL project lead and I wanted to share an official CWL perspective:

The CWL standards include optional use of the Docker format for software containers. It is up to implementers if they want to use the Docker engine, Singularity, udocker, or other engines to run those Docker format containers.

Future versions of the CWL standard will replace the Docker container format with the OCI container image format as that is becoming the new industry standard for software container images.

In the CWL reference runner we currently require the use of the --singularity flag, but in the future we might auto-detect when docker is not available but singularity is.

Cheers,

[adamsla]

@vinjana, Docker does run as root, that is true, but in the LSF configuration, not users have access to the docker group. So, no users have direct access to the docker command. Access to containers is managed by the LSF administrator, and access to all docker options is additionally controlled by the LSF administrator.

To get a primer, please goto the following page: https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_welcome/lsf_kc_containers.html

For a specific use case, review the following: https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_config_ref/lsb.applications.5.html

Search for the keyword "docker-options.sh" for how we can inspect the users submission environment including LSF Job variables to customize the creation of docker options for the application profile.

Again when enabling Docker/Shifter/Singularity/NVidia-Docker for LSF, no user is required to be a part of the Docker groups in Linux.

[vinjana]

@adamsla Thanks for the clarification! Your hints exposed some new facets of the container issue to me and the LSF-support seems to be a good solution to get going with trusted containers and trusted users.

Still, we hope to find a safe and secure solution in which we do not have to administer the container usage while still give our users the freedom to run container-based workflows -- i.e. an untrusted users with untrusted containers setting. We plan to use Singularity for this use case. In the moment, I am a bit unsure, however, whether some kind of support by the workflow management system is reasonable in this case. cwltool has such a support, and because we use LSF and cwlexec is our current favourite CWL workflow management system (Nice code! And JVM! :-) ) I wanted to start a discussion about this.

[adamsla]

Sure, IBM Spectrum LSF Process Manager has a very robust Java API for such things, but compete's a little with CWL at the same time. Our Process Manager tool is very robust with over 15 years of enhancements. However, we will support and continue to support other technologies even when they are somewhat competitive to ours. This is one of the reasons that we are now here on GitHub!

[KerstenBreuer]

Hey there, we would also be highly intressed in the use of singularity in combination with cwlexec. The secure management of docker containers through LSF is a nice feature but it always requires a request from the users to the administrator if a new container is needed. One of the core ideas of dockerized CWL workflows is the "plug-and-play" experience. Singularity appears to have a nearly perfect solution for that. I really hope to see support by cwlexec.

[adamsla]

Singularity is already supported by LSF, you simply need to create the integration. The only change is that you must submit with the required application profile, and if specifying a specific container or version of container, setting an environment variable to specify the container. The application profile holds the repository URL (and any additional docker/singularity) options, and the environment variable allows you to specify the container name and version from that repository. Of course, you can also specify the container name and version in the application profile is you wish.

Check this page for more information.

https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_welcome/lsf_kc_singularity.html

[KerstenBreuer]

Thanks a lot. I will try out that solution and will update you with our experiences.

[vinjana]

O.k. We tried an application profile, but noticed that

• cwlexec composes a Singularity command with --volume parameters, but --volume is invalid for Singularity, where it should be -B!
• the --volume parameters are not forwarded to the actually executed singularity command! Instead Singularity successfully starts, but fails as soon as there is an attempt to access the supposedly mounted directories. The reason may be that LSB_CONTAINER_OPTIONS (https://github.com/IBMSpectrumComputing/cwlexec/blob/d3ff78d7fd8cb4c7e2272f90e2b65b6004719c1a/src/main/java/com/ibm/spectrumcomputing/cwl/exec/util/command/LSFCommandUtil.java#L212) is not an official LSF variable. At least, in the official docs I do find LSB_CONTAINER_IMAGE image, but no LSB_CONTAINER_OPTIONS. Is this maybe an option specifically available in your LSF cluster? In that case, the name LSB_CONTAINER_OPTIONS -- that sounds like an official LSF variable -- is really misleading.