SIGSEGV in kaniko/pkg/util.mkdirAllWithPermissions

[pschulten]

Actual behavior

...
Kaniko version :  v1.5.2
/kaniko/executor $KANIKO_CACHE_ARGS --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --cache-repo $CI_REGISTRY_IMAGE --destination $TAG
Resolved base name quay.io/quarkus/centos-quarkus-maven:21.0.0.2-java11 to build 
Using dockerignore file: /builds/docker/ticker/.dockerignore 
Retrieving image manifest quay.io/quarkus/centos-quarkus-maven:21.0.0.2-java11 
Retrieving image quay.io/quarkus/centos-quarkus-maven:21.0.0.2-java11 from registry quay.io 
Retrieving image manifest quay.io/quarkus/centos-quarkus-maven:21.0.0.2-java11 
Returning cached image manifest              
Retrieving image manifest gcr.io/distroless/static:nonroot 
Retrieving image gcr.io/distroless/static:nonroot from registry gcr.io 
Retrieving image manifest gcr.io/distroless/static:nonroot 
Returning cached image manifest              
Built cross stage deps: map[]                
Retrieving image manifest quay.io/quarkus/centos-quarkus-maven:21.0.0.2-java11 
Returning cached image manifest              
Retrieving image manifest quay.io/quarkus/centos-quarkus-maven:21.0.0.2-java11 
Returning cached image manifest              
Executing 0 build triggers                   
Checking for cached layer docker.signintra.com/docker/ticker:76c89b47a3f4cef7940bb16667931c4b543e707245a75a6586b80cbb5e5647d1... 
No cached layer found for cmd COPY . /ws     
Unpacking rootfs as cmd COPY . /ws requires it. 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x872237]

goroutine 1 [running]:
github.com/GoogleContainerTools/kaniko/pkg/util.mkdirAllWithPermissions(0xc00040aeb0, 0xf, 0x800001fd, 0x0, 0xc, 0x1, 0x1)
    /go/src/github.com/GoogleContainerTools/kaniko/pkg/util/fs_util.go:758 +0x467
github.com/GoogleContainerTools/kaniko/pkg/util.ExtractFile(0x1f91b56, 0x1, 0xc000f2c9a0, 0x23a7dc0, 0xc000296b40, 0x0, 0x0)
    /go/src/github.com/GoogleContainerTools/kaniko/pkg/util/fs_util.go:330 +0xd82
github.com/GoogleContainerTools/kaniko/pkg/util.GetFSFromLayers(0x1f91b56, 0x1, 0xc0002d5160, 0x2, 0x2, 0xc000b0eeb0, 0x1, 0x1, 0x0, 0x0, ...)
    /go/src/github.com/GoogleContainerTools/kaniko/pkg/util/fs_util.go:191 +0x3ae
github.com/GoogleContainerTools/kaniko/pkg/util.GetFSFromImage(0x1f91b56, 0x1, 0x241e740, 0xc0002c2160, 0x207cb40, 0x1, 0x1, 0x4, 0x0, 0x0)
    /go/src/github.com/GoogleContainerTools/kaniko/pkg/util/fs_util.go:127 +0x13b
github.com/GoogleContainerTools/kaniko/pkg/executor.(*stageBuilder).build(0xc00084e840, 0xc00041c21d, 0x5)
    /go/src/github.com/GoogleContainerTools/kaniko/pkg/executor/build.go:305 +0x355
github.com/GoogleContainerTools/kaniko/pkg/executor.DoBuild(0x345a240, 0x1, 0x0, 0x0, 0xc000990000)
    /go/src/github.com/GoogleContainerTools/kaniko/pkg/executor/build.go:594 +0x64c
github.com/GoogleContainerTools/kaniko/cmd/executor/cmd.glob..func2(0x34412c0, 0xc000934000, 0x0, 0xb)
    /go/src/github.com/GoogleContainerTools/kaniko/cmd/executor/cmd/root.go:112 +0xe1
... 

To Reproduce Not easy. We're building our docker images with gitlab-ci (and this works great THANKS!) but it fails with this image (stripped for this issue)

FROM quay.io/quarkus/centos-quarkus-maven:21.0.0.2-java11 AS build
COPY . /ws
FROM gcr.io/distroless/static:nonroot

Additional Information

• building locally works normally with kaniko in docker and docker build
• always fails on gitlab runners regardless of using docker or kubernetes executors (using an image that itself was build with kaniko)
• maybe not really related to kaniko but the fault happends in the kaniko codebase

[pschulten]

I circumvented my issue by using my own graalvm builder image. So this could be closed unless you want to work on the nil pointer

[aokulovx5]

have the same issues, but on 1.6.0 kaniko executor

but i use own kaniko docker image, on alpine, that copy binaries from official kaniko image

in official image (from scratch) issues not repeated, but on my alpine image with kaniko binaries repeated

FROM alpine:3.13
...
COPY --from=gcr.io/kaniko-project/executor:v1.6.0 /kaniko/executor /kaniko/executor
COPY --from=gcr.io/kaniko-project/warmer:v1.6.0 /kaniko/warmer /kaniko/warmer

[ajaykmis]

+1 I am getting the same error both with kaniko v1.5.2 and v1.6.0. @pschulten the other similar issues seem to be due to misaligned address for arm.

How did you fix it? do you have a working dockerfile?

[ajaykmis]

@tejal29 any idea what the runtime panic indicates?

[gewei2000]

+1

[hahait]

+1

[hahait]

The following configuration works normally

FROM gcr.io/kaniko-project/executor:latest AS kaniko

FROM centos:7 

COPY --from=kaniko /kaniko /kaniko

RUN mkdir /workspace /cache -p

ENV PATH=$PATH:/kaniko \
    SSL_CERT_DIR=/kaniko/ssl/certs \
    DOCKER_CONFIG=/kaniko/.docker/

I don't know，if This means that you must use centos system; debain system test can not

[Panlq]

+1 I am getting the same error Kaniko version : v1.6.0 context: 216.3MB docker build success. run ok at kaniko v0.14.0

[zchee]

I don't know the root cause and I haven't dig into it, but the workaround based on @aokulovx5 suggestion, use gcr.io/kaniko-project/executor:v1.6.0-debug should work at least on GCB.

[AgentGoldPaw]

v1.6.0 does not work on amazonlinux:latest. :(

[cmoulliard]

I also experiment such an error using RHEL8 image on k8s. I can reproduce it easily using kind tool See my ticket: https://github.com/redhat-buildpacks/poc/issues/9 and comment: https://github.com/redhat-buildpacks/poc/issues/9#issuecomment-974848272

[tiverive]

Just want to let others know that by adding the following command to the dockerfile I was able to avoid the error in alpine. rm /var/spool/mail

[gewei2000]

Just want to let others know that by adding the following command to the dockerfile I was able to avoid the error in alpine. rm /var/spool/mail

great

[driverxdw]

have the same issues, but on 1.6.0 kaniko executor

but i use own kaniko docker image, on alpine, that copy binaries from official kaniko image

in official image (from scratch) issues not repeated, but on my alpine image with kaniko binaries repeated

FROM alpine:3.13
...
COPY --from=gcr.io/kaniko-project/executor:v1.6.0 /kaniko/executor /kaniko/executor
COPY --from=gcr.io/kaniko-project/warmer:v1.6.0 /kaniko/warmer /kaniko/warmer

Kaniko will decompress the base image and overwrite the original file system. Once the original file system's directories such as [libc, usr, bin, etc.] are overwritten, some operations in the dockerfile will be unavailable, and some commands will show that the so library or some symbols is missing（such as python） , compilation may fail. After the compilation is complete, the file system of the container based on your own image will be destroyed.

[driverxdw]

If you want to customize the image, it is recommended to modify it on the basis of the dockerfile deploy/Dockerfile_debug and use scratch as base image and use busybox file system as default file system. In addition, it is best to specify --cleanup to clean up the unpacked file system directory when compiling with kaniko. If you want to bring some of your own programs into the image, it is best to compile statically, or if the target program is not convenient for static compilation, you can also add the entire / file system（of course you can exclude some files that you really don't use，entire / file system is too huge） of the host/container where the program is located into the image. After compiling the image with kaniko in the container based on your own image, release it（the root file system you brought in） to / , and then set the system environment variables, so that you should be able to run some of your own programs without reporting that libc.so or other something is missing because the file system of the container has been replaced dynamically. I am very sorry for my poor English, but I was deeply troubled by the feature that kaniko can modify the file system before. I would like to help everyone

[Panlq]

this was fixed in https://github.com/GoogleContainerTools/kaniko/pull/1813