Kaniko Build with Gvisor "failed to register layer: link operation not permitted"

[pierrevillard]

Hello,

Actual behavior

I am building OCI images from on EKS cluster with "t3.medium" instances, Containerd and RunC/Gvisor enabled from a Gitlab-CI Runner. I use multi-stage build

This is my build stage:

build:
  stage: build
  image:
    name: ${REGISTRY_HOST}/kaniko-project/executor:v1.17.0-debug  # Also tried executor:v1.16.0-debug
    entrypoint: [""]
  script:
    - /busybox/sh -c "echo {@auths@:{@${REGISTRY_HOST}@:{@auth@:@$(printf "%s:%s" "${REGISTRY_USER}" "${REGISTRY_PASSWORD}" | base64 | tr -d '\n')@}}} > /kaniko/.docker/config.json"
    - /busybox/sh -c "cat /kaniko/.docker/config.json"
    - /busybox/sh -c "sed -i 's/@/\"/g' /kaniko/.docker/config.json"
    - /busybox/sh -c "cat /kaniko/.docker/config.json"
    - /kaniko/executor
      --context "./app/"
      --dockerfile "./app/Dockerfile"
      --destination "registry.hp.cloud-factory.dma.dsit.aws.internal.cloud.edf.fr/clf-cld-factory/flask:v0.0.2"
      --skip-tls-verify
      --force     # Also tryied without --force

With Gvisor OR RunC enabled, build is completed, and no errors in logs, then image is pushed on registry (Harbor). The problem is then i want to PULL images from my running env (On EKS/Containerd, or on my local Desktop with Docker-CE). Images builded with RunC are OK, but images builded with Gvisor can't extract layers after pull...

Logs from Containerd:

Pulling image "XXXXXXXXXXXXX/flask:v0.0.2"
  Warning  Failed     21s (x2 over 54s)  kubelet            Error: ErrImagePull
  Warning  Failed     21s                kubelet            Failed to pull image "XXXXXXXXXXXX/flask:v0.0.2": rpc error: code = Unknown desc = failed to pull and unpack image "XXXXXXXX/flask:v0.0.2": failed to extract layer sha256:58f32bc1a6f6c7d3216d04a43eb8f061cb73dd091a8d0700c1bfe6bbbbd0e1da: link /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/157/fs /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/157/fs/tmp: operation not permitted: unknown

Logs from Docker:

Pulling from XXXXXXXXXXXXXXX/flask
9160faa7ad21: Already exists
9d9d45c409de: Already exists
30f98bb13e13: Already exists
a6ac163cf577: Already exists
905b2cca7598: Already exists
e97f4f915b0f: Already exists
8460253abd63: Already exists
ad5713003ffb: Already exists
bbf9011b8cfb: Already exists
813ce1a041ee: Already exists
baddbb64c904: Already exists
fbab7748ef66: Extracting [==================================================>]  37.29kB/37.29kB
4ebe61cfc663: Download complete
failed to register layer: link / /tmp: operation not permitted

Expected behavior No Pull error for OCI images builded with kaniko on Runc or Gvisor

To Reproduce Steps to reproduce the behavior:

1. Install gvisor following official documentation
2. Laund a kaniko container with gvisor and build an OCI image
3. Push OCI image to a registry
4. Pull the OCI image with Docker or Container, running gvisor or RunC

Does kaniko still support Gvisor builds ?

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [ ]
Please check if this error is seen when you use --cache flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [x]

[romph]

I'm facing the same problem. It would be really nice to find a solution to fix this.

[kamroy]

Hello,

Same problem for me. Anyone has workaround for this issue ?

[pierrevillard]

Any update ?