buildkitd pod is failing on kubernetes cluster having PSP even after allowing unmasked ProcMounTypes

moby / buildkit

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit

https://github.com/moby/moby/issues/34227

Apache License 2.0

8.2k stars 1.16k forks source link

buildkitd pod is failing on kubernetes cluster having PSP even after allowing unmasked ProcMounTypes #1671

Open asmi10 opened 4 years ago

asmi10 commented 4 years ago

Hello!

I tried to deploy buildkitd in kubernetes cluster having below PSP applied, but it failed with error: [rootlesskit:parent] error: failed to setup UID/GID map: newuidmap 10 [0 1000 1 1 100000 65536] failed: newuidmap: Could not set caps : exit status 1 If I allow privilege escalation in PSP then it works fine.

spec: allowPrivilegeEscalation: false allowedCapabilities:

'*' allowedProcMountTypes:
Unmasked fsGroup: ranges:
- max: 65535 min: 1 rule: MustRunAs hostIPC: true hostPID: true hostPorts:
max: 65535 min: 0 runAsUser: rule: RunAsAny seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes:
'*'

nickhs commented 3 years ago

I'm having this issue as well - a few notes:

newuidmap has the setuid bit but allowPrivilegeEscalation: false prevents that from happening. Even if filecaps were used the same issue would happen
It's not clear to me why newuidmap needs setuid? I can do unshare -r (which writes to /proc/pid/uid_map) without needing CAP_SETUID.
newuidmap calls capset here but again its not totally clear to me why it needs to do so?
By allowing priv escalation doesn't that imply the container can become root if run from an untrusted image?

AkihiroSuda commented 3 years ago

It's not clear to me why newuidmap needs setuid? I can do unshare -r (which writes to /proc/pid/uid_map) without needing CAP_SETUID.

For writing multiple entries to uid_map, newuidmap needs setuid

candrews commented 1 month ago

Is there any way to eliminate the use of newuidmap/newgidmap?

I need to be able to run buildkit in a context with allowPrivilegeEscalation: false.

I came across https://unix.stackexchange.com/questions/759533/what-causes-newuidmap-to-be-dissallowed-in-new-user-namespace - perhaps it provides a way forward?

AkihiroSuda commented 1 month ago

Is there any way to eliminate the use of newuidmap/newgidmap?

I need to be able to run buildkit in a context with allowPrivilegeEscalation: false.

Conceptually yes, by using UML as in https://github.com/weber-software/diuid

Or just use seccomp_unotify to hook UID-relevant syscalls (POC https://github.com/rootless-containers/subuidless)

candrews commented 1 month ago

Those projects are unmaintained - is there a maintained, actively used solution available?