Singularity running as root inside an unprivileged Kubernetes pod fails to extract images

[stxue1]

Version of Singularity What version of Singularity are you using? Run:

$ singularity --version
singularity-ce version 4.1.2-jammy

Describe the bug Extraction with unsquashfs 4.5 in a unprivileged Kubernetes Pod with UID 0 can cause an extraction failure.

write_xattr: failed to write xattr security.selinux for file /image/root/.exec because extended attributes are not supported by the destination filesystem

Ignoring xattrs in filesystem

To avoid this error message, specify -no-xattrs
Parallel unsquashfs: Using 2 processors
25 inodes (22 blocks) to write

[==============================================================|   ] 21/22  95%

created 19 files
created 14 directories
created 5 symlinks
created 0 devices
created 0 fifos
created 0 sockets
DEBUG   [U=0,P=3041]       Master()                      Child exited with exit status 2
: exit status 2

To Reproduce Steps to reproduce the behavior: In a Kubernetes cluster:

kubectl run -t ubuntu --image=ubuntu:22.04 -i
# rest runs in the launched shell
apt update && apt install wget && wget https://github.com/sylabs/singularity/releases/download/v4.1.2/singularity-ce_4.1.2-jammy_amd64.deb && apt-get install ./singularity-ce_*.deb
touch /etc/localtime
singularity -d -v run docker://hello-world

Logs

Expected behavior Singularity should detect that even though it's running as a root user, it does not have the permissions to apply xattr's that it wants to

OS / Linux Distribution

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Installation Method Installed from the Singularity github release

Additional context With the --privileged flag, Singularity is able to use the xattr's it wants and is able to run properly. It doesn't seem to matter if I add the -u flag to the singularity command, as singularity run -u docker://hello-world fails similarly.

[dtrudg]

First, a note that if this specific issue is fixed, you'll still hit problems further on. Singularity is going to need to create namespaces, perform mount syscalls etc. later on in container execution. You will need to grant enough privileges for this to happen.

We don't often see people attempting to run inside kubernetes, and don't actively test this workflow. However, you may find some useful information in the podman documentation page linked below... Singularity is going to require broadly the same privileges as other runtimes. It is not possible to run a container runtime inside a pod without granting any additional privileges.

https://www.redhat.com/sysadmin/podman-inside-kubernetes

As an aside - it'd be good to understand the workflow here. Given that you can run an OCI container natively under kubernetes, what are the circumstances that require you to run under singularity inside k8s?

With regard specifically to the detection of the ability to apply xattrs - we already attempt to detect a rootless (unprivileged) situation, and filesystems that do not support setting xattrs:

https://github.com/sylabs/singularity/blob/74feead7af2550cf34ad0e2fbc81415acdefe770/internal/pkg/image/unpacker/squashfs.go#L110

Clearly this is not working in the kubernetes case. If you are able to experiment with the detection process to identify the cause then we'd welcome a patch, or further information that would allow us to create a patch for this situation.