[OKD] Permission denied to mount "proc" to rootfs at "/proc" with moby/buildkit:v0.9.3-rootless on Kubernetes

[mbarbero]

I run buildkitd on a Kubernetes cluster (OKD 4.7, the "upstream" FLOSS of Openshift). The daemon starts and stays up. But every time I try to run a build, I get an error:

❯ ./bin/buildctl \
  --addr=tcp://127.0.0.1:1234 \
  build \
  --frontend=dockerfile.v0 \
  --local context="." \
  --local dockerfile="." \
  --output "type=image,\"name=testbuildctl-1\""
[+] Building 1.2s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                     0.4s
 => => transferring dockerfile: 74B                                                                                                                                                                      0.4s
 => [internal] load .dockerignore                                                                                                                                                                        0.3s
 => => transferring context: 2B                                                                                                                                                                          0.3s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                         0.5s
 => CACHED [1/2] FROM docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300                                                                                   0.0s
 => => resolve docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300                                                                                          0.0s
 => ERROR [2/2] RUN echo "Hello world"                                                                                                                                                                   0.1s
------
 > [2/2] RUN echo "Hello world":
#5 0.072 container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "proc" to rootfs at "/proc" caused: mount through procfd: permission denied
------
Dockerfile:3
--------------------
   1 |     FROM alpine
   2 |
   3 | >>> RUN echo "Hello world"
   4 |
   5 |
--------------------
error: failed to solve: process "/bin/sh -c echo \"Hello world\"" did not complete successfully: exit code: 1

Dockerfile is very simple:

FROM alpine

RUN echo "Hello world"

As I'm on OKD, I created a dedicated Security Context Constraint in order to not drop the SETGID and SETUID capabilities as it does by default and also to allow the unconfined seccomp profile. This SCC can be read here.

The deployment definition is largely based on the example provided by buildkit. Note that I dropped the option --oci-worker-no-process-sandbox and added procMount: "Unmasked" to the pod security context in my deployment. I also tried the other way around, it does not change anything regarding the error mentioned above.

I also tried running a privileged buildkitd (using the moby/buildkit:master image, privileged: true for the security context, and added the privileged SCC to the service account running the pod. Everything works as expected: I can build my container image from the Dockerfile without any issue.

The last attempt I made is to run the rootless image with the privileged SCC, but it does not work either. I get the same build error. My guess is that something is missing on the deployment configuration, but I'm running out of idea.

Am I missing anything for running rootless buildkitd on OKD/Openshift?

Thanks.

---

Software versions: Server Version: 4.7.0-0.okd-2021-07-03-190901 Kubernetes Version: v1.20.0-1085+01c9f3f43ffcf0-dirty

[AkihiroSuda]

Could you try setting securityContext.seLinuxOptions.type to spc_t https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.23/#selinuxoptions-v1-core

[AkihiroSuda]

Potentially related: https://github.com/moby/buildkit/issues/1634

[mbarbero]

Thanks for your reply and suggestion. Unfortunately, I get the same error with securityContext.seLinuxOptions.type set to spc_t.

It seems that a "recent" change to runc added some more validation of the mount destination. The way proc is mounted seems to break here.

FYI, here is the output of runc --version on my hosts.

$ runc -version
runc version 1.0.0-rc95
commit: 4c62ef789fd7a2963bf61ffbf421ce9646063648
spec: 1.0.2-dev
go: go1.16.3
libseccomp: 2.5.0

[AkihiroSuda]

Does moby/buildkit:master-rootless work?

[mbarbero]

Does moby/buildkit:master-rootless work?

Both moby/buildkit:master-rootless and moby/buildkit:rootless fail with the same error

I did not try again master-rootless since I reported the issue though. Are there changes that could have fixed the issue and you want me to try?

[r351574nc3]

Did anyone solve this? This has been nearly 9 months.

[mbarbero]

Did anyone solve this? This has been nearly 9 months.

No, this is still an issue for us.

[AkihiroSuda]

Does https://github.com/moby/buildkit/pull/3203 work?