Support Singularity?

[biocyberman]

For security and management reasons, we want to use Singularity instead of Docker. The difficulty with Singularity for me right now is that little documentation exists about how to run a container with Desktop environment. It's something our users want to use. Since there are similarities between Docker and Singularity, would it be possible to adapt x11docker for Singularity? I can also hack and write some code, but from the first look of 8k+ lines of code, I think it is better to ask you first @mviereck .

[mviereck]

Since there are similarities between Docker and Singularity, would it be possible to adapt x11docker for Singularity?

I'd be interested to support singularity, too. However I am a noob here, and have no experience with singularity at all. Singularity recently appeared in the debian repository, so I can run some tests.

I can also hack and write some code

Can you point me to a beginner's introduction? Or can you provide me a few lines of code to get a container running? If you can introduce me a bit into singularity and help me above some hurdles, we might be able to adapt x11docker together.

For security and management reasons, we want to use Singularity instead of Docker.

Where do you see a security advantage compared to Docker?

The difficulty with Singularity for me right now is that little documentation exists about how to run a container with Desktop environment.

Basically it would be same as with docker: share a unix socket and provide X access credentials. Compare: https://github.com/mviereck/x11docker/wiki/Short-setups-to-provide-X-display-to-container. If sharing a unix socket is not possible for some reasons, an X over TCP setup should work. At least a basic setup should not be too hard.

[biocyberman]

Singularity recently appeared in the debian repository, so I can run some tests.

Haven't check the version on Debian, I would recommend to compile and install the latest stable version from github (see link to the documentation in the end). From version 3.3, Singularity supports fakeroot feature, which means users do not need sudo right to build from a Definition file

Where do you see a security advantage compared to Docker?

We first tried docker for NVIDIA DGX2 server in a multiuser environment setup. One of the big deal breaker there is that docker doesn't respect SLURM job specifications of number of CPU. That gives a user unconstrained access to all available GPUs. I read somewhere that one may have to enable some kernel feature and recompile the kernel so Docker will respect cgroup settings. That is not something we want to do. Singularity on the other hand respects SLURM job specification. Singularity is developed with HPC+multiuser environment in mind.

Another problem, to run Docker, one must be in Docker group, or worse, to have sudo right. This is another deal breaker. I tried with x11docker but this seems only secure if an admin starts a container and give access to the normal users. User should not be able to build image or start a container, otherwise the virtual root in the container may do privilege escalation to and effect things outside the container.

All in all, I see there are to many tweaks to do, that they both takes time and introduce users to more complex workflow.

For complete picture, NVIDIA is also developing something similar to Singularity but it uses Docker images after conversion. That is it doesn't have an Dockerfile-like: https://github.com/NVIDIA/enroot

Can you point me to a beginner's introduction?

I recommend singularity's own documentation for all things but the GUI desktop environment :) https://sylabs.io/guides/3.4/user-guide/quick_start.html

You may also watch this slow pace beginner guide for usage of Singularity https://www.youtube.com/watch?v=gev_nIw02u4&list=PLO7bpxApY95TlTY7pMyPIk0RaJQD85NnG&index=4

share a unix socket and provide X access credentials. Compare: https://github.com/mviereck/x11docker/wiki/Short-setups-to-provide-X-display-to-container. If sharing a unix socket is not possible for some reasons, an X over TCP setup should work. I will check this out while waiting for you :)

[mviereck]

A quick first success! By default singularity shares /tmp from host. This includes /tmp/.X11-unix/ with the X unix sockets from host. This allows X access

• Authenticate X access with xhost (run on host): xhost +SI:localuser:$USER
• Get a GUI image: singularity pull docker://x11docker/lxde
• Run singularity image:
• singularity shell lxde_latest.sif
• Run GUI: DISPLAY=:0 lxterminal

I'll reply to your post a bit later in detail, just wanted to tell this quickly. :-) This setup is unpolished and insecure, but already a proof of concept.

btw: These days I have some internet connection issues, so I might be slow to respond until I'm online again.

[biocyberman]

I am not familiar with xhost, how do I add a Kerberos user instead of localuser? I checked man xhost and tried: xhost +SI:krb:$USER And got this:

krb:user@example.com being added to access control list
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  109 (X_ChangeHosts)
  Value in failed request:  0x12
  Serial number of failed request:  7
  Current serial number in output stream:  9

[mviereck]

I am not familiar with xhost, how do I add a Kerberos user instead of localuser?

ok, I see, it is some sort of network setup. I am not familar with network setups in general.

There are two ways to allow access to X, either with xhost or with a cookie in $XAUTHORITY. Let's find out how your regular X access is done.

• Please check the output of plain xhost if there is already an useable entry.
• Check echo $XAUTHORITY if there is a cookie file. If XAUTHORITY is set and the file is already available in singularity, try:

  DISPLAY=:0 XAUTHORITY=/path/to/cookie lxterminal

  Check also if DISPLAY on host is really :0 or :0.0.

X Error of failed request: BadValue (integer parameter out of range for operation)

I am not sure why xhost gives this error. It is not a syntax error, but some X11 protocol error. Maybe the user has an UID out of some integer range?

[mviereck]

Does singularity not run on your local system at all but on a remote server without a display? In that case you need to allow access to your local display. If you access the server with ssh, try with ssh -X and check out DISPLAY in the ssh session. DISPLAY should look like <IP-adress>:10. In that case no unix socket is involved.

[mviereck]

We first tried docker for NVIDIA DGX2 server in a multiuser environment setup. One of the big deal breaker there is that docker doesn't respect SLURM job specifications of number of CPU. That gives a user unconstrained access to all available GPUs.

docker has a new option --gpus to specify which NVIDIA GPUs and which of its capabilities can be used by the container. That might be partially what you need. Compare https://github.com/docker/cli/issues/1200

Another problem, to run Docker, one must be in Docker group, or worse, to have sudo right. This is another deal breaker.

Yes, that is quite unfortunate. podman might be an alternative for containers without root, I am waiting for a debian package: https://github.com/containers/libpod

I tried with x11docker but this seems only secure if an admin starts a container and give access to the normal users.

x11docker does its best to isolate untrusted applications, but to run docker the user must be trusted. That is a no-go in a multi-user environment. However, there is one possibility to run docker or x11docker without giving the user root permissions or membership in group docker:

• The admin creates a launcher script, e.g. /usr/local/bin/x11docker-xfce:

  #! /bin/bash  
  x11docker --desktop x11docker/xfce

• Make an entry in /etc/sudoers that allows the user to run this command with sudo and without a password. The user does NOT need to be member of group sudo.

  someuser ALL=NOPASSWD:/usr/local/bin/x11docker/xfce

  Though, this user cannot pull or build images. That would need further setup.

Haven't check the version on Debian, I would recommend to compile and install the latest stable version from github (see link to the documentation in the end). From version 3.3, Singularity supports fakeroot feature, which means users do not need sudo right to build from a Definition file

Debian currently ships with singularity version 3.1.1, but I think that is ok for now. It is enough to check out X setups, and I'll use the GUI images I've developed for docker.

security advantage compared to Docker

Some first thoughts about the security setup of singularity: So far I think the only (but important) security advantage of singularity is that the user is not in group docker. I see disadvantages: The user is not protected at all. By default, entire HOME is shared. If a malicious application is installed with singularity, it can spy passwords, steal data, , change ~/.bashrc etc. pp. It only does not have root privileges. Also entire /dev seems to be shared. I am not sure if that is a good idea.

Only entirely trusted images should be used with singularity. But the concept allows to run arbitrary images found somewhere in the net.

[biocyberman]

Only entirely trusted images should be used with singularity. But the concept allows to run arbitrary images found somewhere in the net.

Wouldn't this drawback applied for all programs that users have direct SSH access to host, including Docker images? A solution then is a web interface with prebuild images that users can launch. If they want a customized image, they have to send build requests. That then will sounds a lot like Cloud technologies like Azure, Google, AWS, or the self-hosted Openstack.

Thanks and agree with you about other things.

[biocyberman]

I tried and could start a clean lxterminal and mate-terminal (with the mate image).

user@clientlaptop:~/ ssh -X user@front-end
user@front-end:~ singularity pull docker://x11docker/mate
user@front-end:~ mkdir runuser
user@front-end:~ srun --pty --gres=gpu:1 --x11 singularity shell --nv -B ./runuser:/run/user/$UID mate_latest.sif
Singularity mate_latest.sif:~/singularity> mate-terminal # terminal came up clean on my laptop
Singularity mate_latest.sif:~/singularity> caja 
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
#  but still could come up on my laptop

Problems come when I tried a full desktop environment:

mate-session
Failed to run gnome-keyring-daemon: Failed to execute child process “gnome-keyring-daemon” (No such file or directory)
mate-session[90394]: WARNING: Could not connect to ConsoleKit: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
mate-session[90394]: WARNING: Could not connect to ConsoleKit: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
mate-session[90394]: WARNING: Unable to find provider '' of required component 'dock'
Window manager warning: Screen 0 on display "localhost:73.0" already has a window manager
[1570186467,000,xklavier.c:xkl_engine_start_listen/]    The backend does not require manual layout management - but it is provided by the application
Window manager warning: Screen 0 on display "localhost:73.0" already has a window manager

(mate-panel:90445): GVFS-RemoteVolumeMonitor-WARNING **: 12:54:28.312: remote volume monitor with dbus name org.gtk.vfs.UDisks2VolumeMonitor is not supported
Unable to open desktop file /usr/share/applications/firefox.desktop for panel launcher: No such file or directory
mate-session[90394]: WARNING: Could not connect to ConsoleKit: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
mate-session[90394]: WARNING: Could not connect to ConsoleKit: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
mate-session[90394]: WARNING: Could not connect to ConsoleKit: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
mate-session[90394]: WARNING: Could not connect to ConsoleKit: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

(caja:90457): GVFS-RemoteVolumeMonitor-WARNING **: 12:54:28.804: remote volume monitor with dbus name org.gtk.vfs.UDisks2VolumeMonitor is not supported
dbus-daemon[90507]: Activating service name='org.a11y.atspi.Registry' requested by ':1.0' (uid=140196 pid=90489 comm="/usr/lib/mate-applets/trashapplet " label="unconfined")
dbus-daemon[90507]: Successfully activated service 'org.a11y.atspi.Registry'
SpiRegistry daemon is running with well-known name - org.a11y.atspi.Registry

(mate-settings-daemon:90420): GVFS-RemoteVolumeMonitor-WARNING **: 12:54:29.106: remote volume monitor with dbus name org.gtk.vfs.UDisks2VolumeMonitor is not supported

(polkit-mate-authentication-agent-1:90497): polkit-mate-1-WARNING **: 12:54:29.130: Error getting authority: Error initializing authority: Could not connect: No such file or directory

** (at-spi2-registryd:90514): WARNING **: 12:54:29.199: Failed to register client: GDBus.Error:org.gnome.SessionManager.AlreadyRegistered: Unable to register client

** (at-spi2-registryd:90514): WARNING **: 12:54:29.199: Unable to register client with session manager

** (mate-settings-daemon:90420): WARNING **: 12:54:29.342: Clipboard manager is already running.

** (caja:90457): WARNING **: 12:54:29.494: Can not calculate _NET_NUMBER_OF_DESKTOPS

** (caja:90457): WARNING **: 12:54:29.494: Can not calculate _NET_NUMBER_OF_DESKTOPS

** (caja:90457): WARNING **: 12:54:29.494: Can not get _NET_WORKAREA

** (caja:90457): WARNING **: 12:54:29.494: Can not determine workarea, guessing at layout

^Cmate-session[90394]: WARNING: Client '/org/gnome/SessionManager/Client6' failed to reply before timeout
mate-session[90394]: WARNING: Client '/org/gnome/SessionManager/Client1' failed to reply before timeout
mate-session[90394]: WARNING: Client '/org/gnome/SessionManager/Client7' failed to reply before timeout
srun: error: _half_duplex: wrote -1 of 4096
Gdk-Message: 12:56:25.473: mate-session: Fatal IO error 4 (Interrupted system call) on X server localhost:73.0. 

Using fakeroot

srun --pty --gres=gpu:1 --x11 singularity shell --fakeroot --dns 8.8.8.8 --net --network bridge,ptp --nv -B ./runuser:/run/user/$UID mate_latest.sif
srun: job XXXX queued and waiting for resources
srun: job XXXX has been allocated resources
INFO:    Convert SIF file to sandbox...
WARNING: underlay of /usr/bin/nvidia-smi required more than 50 (465) bind mounts
Singularity mate_latest.sif:/home/user/test_singularity> mate-session
Unable to init server: Could not connect: Connection refused

** (mate-session:96267): WARNING **: 13:18:03.293: Cannot open display:
 apt-get update
Hit:1 http://deb.debian.org/debian buster InRelease
Hit:2 http://security.debian.org/debian-security buster/updates InRelease
Hit:3 http://deb.debian.org/debian buster-updates InRelease
Reading package lists... Done
E: Release file for http://security.debian.org/debian-security/dists/buster/updates/InRelease is expired (invalid since 80d 5h 19min 11s). Updates for this repository will not be applied.
E: Release file for http://deb.debian.org/debian/dists/buster-updates/InRelease is expired (invalid since 80d 2h 51min 13s). Updates for this repository will not be applied.
Singularity mate_latest.sif:/home/user/test_singularity> ping google.com
PING google.com (108.177.119.102) 56(84) bytes of data.
64 bytes from 108.177.119.102 (108.177.119.102): icmp_seq=1 ttl=40 time=18.6 ms
^C
--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 18.624/18.624/18.624/0.000 ms

[mviereck]

Wouldn't this drawback applied for all programs that users have direct SSH access to host

Of course, a user can always damage everything he has access to. He can download arbitrary binaries or scripts from the net and execute them. However, not everything is available from secure repositories, and not every dependency can be fulfilled easily.. Containers help here.

It is a question of the defaults:

• Docker:

Docker runs containers with fairly secure defaults. A malicious application will have a hard time to break out of docker run evil/image although it has root permissions in the container.

Docker allows to degrade container isolation with several options. An evil image can do arbitrary harm to the system with e.g. docker run -v /:/hostroot --cap-add=ALL evil/image. It can harm the user space with docker run -v $HOME:$HOME evil/image.

A quite good wrapper could be designed to allow harmless docker options only.

• Singularity

Singularity protects the root system, but shares entire HOME by default. (And also at least /tmp and /dev). This is a comfortable, but quite insecure default. Running a singularity container one time is enough to install a trojan in user space. The problem is: the user is encouraged to to run just everything like on MS Windows, and he'll get the same problems as on MS Windows. And he might think: "hey, it's a container, it is secure."

I prefer secure defaults that can be reduced optionally. For singularity, I'd recommend a default that does not allow to share entire HOME but only subdirectories specified by the user. Also /tmp and /dev and others should only be shared on demand. That would already improve user space security a lot.

A solution then is a web interface with prebuild images that users can launch. If they want a customized image, they have to send build requests.

The user should be aware of the risks. There might be a repository "entirely trusted", another one "basically trusted", and the rest is "unknown wild beast". At least the "unknown wild beast" should have quite restricted defaults.

[biocyberman]

For singularity, I'd recommend a default that does not allow to share entire HOME but only directories specified by the user. Also /tmp and /dev and maybe others should only be shared on demand. That would already improve user space security a lot.

This can be done by configuring singularity.conf. Mine is at /usr/local/etc/singularity/singularity.conf

# MOUNT PROC: [BOOL]
# DEFAULT: yes
# Should we automatically bind mount /proc within the container?
mount proc = yes

# MOUNT SYS: [BOOL]
# DEFAULT: yes
# Should we automatically bind mount /sys within the container?
mount sys = yes

# MOUNT DEV: [yes/no/minimal]
# DEFAULT: yes
# Should we automatically bind mount /dev within the container? If 'minimal'
# is chosen, then only 'null', 'zero', 'random', 'urandom', and 'shm' will
# be included (the same effect as the --contain options)
mount dev = yes

# MOUNT DEVPTS: [BOOL]
# DEFAULT: yes
# Should we mount a new instance of devpts if there is a 'minimal'
# /dev, or -C is passed?  Note, this requires that your kernel was
# configured with CONFIG_DEVPTS_MULTIPLE_INSTANCES=y, or that you're
# running kernel 4.7 or newer.
mount devpts = yes

# MOUNT HOME: [BOOL]
# DEFAULT: yes
# Should we automatically determine the calling user's home directory and
# attempt to mount it's base path into the container? If the --contain option
# is used, the home directory will be created within the session directory or
# can be overridden with the SINGULARITY_HOME or SINGULARITY_WORKDIR
# environment variables (or their corresponding command line options).
mount home = yes

[mviereck]

I tried and could start a clean lxterminal and mate-terminal (with the mate image).

Great!

Problems come when I tried a full desktop environment:

I am not sure what causes the crash. One singularity error is shown:

srun: error: _half_duplex: wrote -1 of 4096

Though, a desktop should normally not be started without an additional X server. Otherwise it interferes with your already running desktop. You can try nested X server Xephyr and run the mate desktop in Xephyr. x11docker can help you to set up Xephyr or other X servers. Run e.g.:

x11docker --xephyr --wm=none --showenv

This will open a Xephyr window and show DISPLAY and XAUTHORITY on stdout.

---

In your fakeroot setup:

Unable to init server: Could not connect: Connection refused

Please check DISPLAY and XAUTHORITY and make sure that the file in XAUTHORITY is available.

---

Probably of interest is GPU support, too. I did a test here, with my local singularity installation GPU acceleration works.

In an ssh setup this becomes difficult. With ssh -X only software rendering is possible. Instead, the remote server has to run an Xorg server or a Wayland compositor (+ Xwayland) to allow GPU usage. This can be forwarded e.g. with VNC or xpra.

Another issue are NVIDIAs closed source drivers. Docker containers need the same NVIDIA driver version in image as on host. Probably singularity will have the same problem. I cannot test that myself, I don't have NVIDIA hardware. Setups with open source drivers (MESA) are an easy task.

[mviereck]

About x11docker and singuarity: Although technically possible, I think it does not make sense to include singularity support in x11docker.

x11docker aims to isolate applications as good as possible. It follows the principle of least privilege. singularity aims to isolate applications as few as possible to avoid hurdles for users. It gives a lot of privileges by default.

Maybe x11docker could drop those privileges with command line options regardless of system wide settings. But it would work against the intentions of singularity. So this does not make sense.

I'd rather like to support podman when it is available as a debian package. podman allows to set up containers without root privileges, but is otherwise similar to docker in its default setup.

---

You can still use x11docker to run an X server and provide this X server to singularity applications. Setting up X on a remote server with GPU support is out of scope of x11docker, but if you want to set this up, I'm happy to help with some hints.

Closing now, but feel free to ask further questions.