genuinetools / img

Standalone, daemon-less, unprivileged Dockerfile and OCI compatible container image builder.
https://blog.jessfraz.com/post/building-container-images-securely-on-kubernetes/
MIT License
3.88k stars 230 forks source link

Official docker image doesn't work with overlayfs backend in kubernetes #307

Closed bbkgh closed 3 years ago

bbkgh commented 3 years ago

Hi. I am running latest img docker image as kubernetes pod with this manifest:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    container.apparmor.security.beta.kubernetes.io/build: unconfined
    container.seccomp.security.alpha.kubernetes.io/build: unconfined
  labels:
    pod: img-test
  name: img-test
  namespace: img-test
spec:
  containers:
    - command:
        - sh
        - -c
        - sleep 4h
      image: r.j3ss.co/img
      imagePullPolicy: Always
      name: build
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        privileged: false

But i cannot use overlayfs as backend and this error happens:

~ $ img pull -b overlayfs nginx
Pulling nginx...
Error: failed to mount /run/user/1000/containerd-mount196575642: invalid argument

I've tested buildkit rootless image with ~same pod manifest (and same node/kernel) and it works correctly with overlayfs :

apiVersion: v1
kind: Pod
metadata:
  name: buildkitd
  annotations:
    container.apparmor.security.beta.kubernetes.io/buildkitd: unconfined
    container.seccomp.security.alpha.kubernetes.io/buildkitd: unconfined
# see buildkit/docs/rootless.md for caveats of rootless mode
spec:
  containers:
  - name: buildkitd
    image: moby/buildkit:master-rootless
    args:
    - --oci-worker-no-process-sandbox
    readinessProbe:
      exec:
        command:
        - buildctl 
        - debug 
        - workers
      initialDelaySeconds: 5
      periodSeconds: 30
    livenessProbe:
      exec:
        command:
        - buildctl 
        - debug 
        - workers
      initialDelaySeconds: 5
      periodSeconds: 30
    securityContext:
# To change UID/GID, you need to rebuild the image
      runAsUser: 1000
      runAsGroup: 1000

kubectl logs buildkitd shows time="2020-11-03T16:35:14Z" level=info msg="auto snapshotter: using overlayfs" and buildctl build command works with overlayfs.

~ $ uname -a
Linux img-test 5.4.0-48-generic #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020 x86_64 Linux
~ $ uname -a
Linux img-test 5.4.0-48-generic #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020 x86_64 Linux
~ $ img version
img:
 version     : v0.5.10
 git hash    : cd9273d6
 go version  : go1.13.15
 go compiler : gc
 platform    : linux/amd64
runc:
 version     : 1.0.0-rc10+dev
 commit      : 56aca5aa50d07548d5db8fd33e9dc562f70f3208
 spec        : 1.0.2

Do you have any idea about this problem? Is there any differences between official image and buildkit-rootless image that can be root of this problem? Thanks