tspearconquest
commented
2 years ago Interestingly,
The below dockerfile seems to get past the issue, but it has other issues:
FROM scratch as one
RUN --mount=from=docker.io/busybox:musl,dst=/usr/ ["/busybox/busybox", "sh", "-c", "/busybox/mkdir -p /kaniko && /busybox/chmod 777 /kaniko"]Which results in the following (incorrect) output:
$ docker run --rm -it --entrypoint "" -v "$(pwd)":/workspace test:1 /bin/sh
/workspace # /kaniko/executor --dockerfile Dockerfile --destination kanikotest:1 --no-push --tarPath test.tar
INFO[0000] Resolved base name scratch to one
INFO[0000] Using dockerignore file: /workspace/.dockerignore
INFO[0000] No base image, nothing to extract
INFO[0000] Built cross stage deps: map[]
INFO[0000] No base image, nothing to extract
INFO[0000] Executing 0 build triggers
INFO[0000] Building stage 'scratch' [idx: '0', base-idx: '-1']
INFO[0000] Unpacking rootfs as cmd RUN --mount=from=docker.io/busybox:musl,dst=/usr/ ["/busybox/busybox", "sh", "-c", "/busybox/mkdir -p /kaniko && /busybox/chmod 777 /kaniko"] requires it.
INFO[0000] RUN --mount=from=docker.io/busybox:musl,dst=/usr/ ["/busybox/busybox", "sh", "-c", "/busybox/mkdir -p /kaniko && /busybox/chmod 777 /kaniko"]
INFO[0000] Initializing snapshotter ...
INFO[0000] Taking snapshot of full filesystem...
INFO[0000] Cmd: /busybox/busybox
INFO[0000] Args: [sh -c /busybox/mkdir -p /kaniko && /busybox/chmod 777 /kaniko]
INFO[0000] Running: [/busybox/busybox sh -c /busybox/mkdir -p /kaniko && /busybox/chmod 777 /kaniko]
INFO[0000] Taking snapshot of full filesystem...
INFO[0000] No files were changed, appending empty layer to config. No layer added to image. <--- this is incorrect, we just created /kaniko
INFO[0000] Skipping push to container registry due to --no-push flagGiven that the busybox container doesn't have a /busybox directory nor a /kaniko directory, and the executor debug container does, it seems like executor is mounting the container's root directory in each of the build stages, and I suspect the RUN --mount=from=busybox command may be being overmounted by the executor container's own root directory.
The mkdir -p flag causes mkdir to return a successful exit code even though the /kaniko directory already exists. So if executor is mounting / from the container into the build stages, then running mkdir /kaniko without the -p flag in a dockerfile should fail no matter what image is being built. So by specifying mkdir -p to create /kaniko to ignore the existing directory, a big issue is avoided, however then snapshotting takes place and the mounted filesystem from executor is no longer mounted. Since that filesystem was a transient filesystem only mounted during build and not leaving anything laying around, the /kaniko directory that was "created" actually disappears and so nothing has actually changed in the filesystem for snapshotting because /kaniko from the executor container already existed and had 0777 permissions on the mounted filesystem from the container.
More evidence can be found in that the below dockerfile fails to build with Kaniko due to /busybox/[ (which is a busybox binary) being in use when COPY --from=busybox /bin /busybox runs in the scratch container near the end of the debug image build. This is essentially the Dockerfile_debug file with only slight tweaks based on the knowledge provided above about executor mounting / from its container into the build stages:
# Copyright 2018 Google, Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
FROM golang:1.17
WORKDIR /src
# This arg is passed by docker buildx & contains the target CPU architecture (e.g., amd64, arm64, etc.)
ARG TARGETARCH
ENV GOARCH=$TARGETARCH
ENV CGO_ENABLED=0
ENV GOBIN=/usr/local/bin
# Get GCR credential helper
RUN go install github.com/GoogleCloudPlatform/docker-credential-gcr@4cdd60d0f2d8a69bc70933f4d7718f9c4e956ff8
# Get Amazon ECR credential helper
RUN go install github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login@69c85dc22db6511932bbf119e1a0cc5c90c69a7f # v0.6.0
# Get ACR docker env credential helper
RUN go install github.com/chrismellard/docker-credential-acr-env@09e2b5a8ac86c3ec347b2473e42b34367d8fa419
# Add .docker config dir
RUN mkdir -p /kaniko/.docker
COPY . .
RUN \
--mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/go/pkg \
make GOARCH=$TARGETARCH && \
make GOARCH=$TARGETARCH out/warmer
# Generate latest ca-certificates
FROM debian:bullseye-slim AS certs
RUN apt update && apt install -y ca-certificates
# use musl busybox since it's statically compiled on all platforms
FROM busybox:musl as busybox
FROM scratch
# Create kaniko directory with world write permission to allow non root run
##### Note that we're using a full path for the busybox binary here but it's not from /usr like it should be.
##### This is the only change to this whole file.
##### We do this because executor is mounting the root directory from the parent debug container over the top of / in ##### the containers that are defined in the Dockerfile being built by executor.
##### In other words, anything in /usr from the RUN --mount flag below is invisible inside this stage due to
##### using executor, and so we have to use the paths from the executor container's root directory to access binaries.
##### In a docker build, this works without fully qualified paths, and in fact, you can use the fully qualified path of the
##### busybox binary by calling for /usr/bin/busybox instead of /busybox/busybox.
##### Additionally, in that case, the entirety of /bin from the busybox stage is accessible so mkdir and chmod work
##### without their respective full paths as well.
RUN --mount=from=busybox,dst=/usr/ ["/busybox/busybox", "sh", "-c", "/busybox/mkdir -p /kaniko && /busybox/chmod 777 /kaniko"]
COPY --from=0 /src/out/executor /kaniko/executor
COPY --from=0 /src/out/warmer /kaniko/warmer
COPY --from=0 /usr/local/bin/docker-credential-gcr /kaniko/docker-credential-gcr
COPY --from=0 /usr/local/bin/docker-credential-ecr-login /kaniko/docker-credential-ecr-login
COPY --from=0 /usr/local/bin/docker-credential-acr-env /kaniko/docker-credential-acr-env
##### This is where we fail when using full paths above.
##### If I change this to copy /bin to /bbox, then the build completes.
##### However, if I then try to build this same Dockerfile (with /busybox changed to /bbox below still)
##### Then the build again fails because /bbox/[ then is in use because the new executor container
##### mounts / over the build stage's root directory still and so /bbox already exists
COPY --from=busybox /bin /busybox
# Declare /busybox as a volume to get it automatically in the path to ignore
VOLUME /busybox
COPY --from=certs /etc/ssl/certs/ca-certificates.crt /kaniko/ssl/certs/
COPY --from=0 /kaniko/.docker /kaniko/.docker
COPY files/nsswitch.conf /etc/nsswitch.conf
ENV HOME /root
ENV USER root
ENV PATH /usr/local/bin:/kaniko:/busybox
ENV SSL_CERT_DIR=/kaniko/ssl/certs
ENV DOCKER_CONFIG /kaniko/.docker/
ENV DOCKER_CREDENTIAL_GCR_CONFIG /kaniko/.config/gcloud/docker_credential_gcr_config.json
WORKDIR /workspace
RUN ["/busybox/mkdir", "-p", "/bin"]
RUN ["/busybox/ln", "-s", "/busybox/sh", "/bin/sh"]
ENTRYPOINT ["/kaniko/executor"]
dradetsky
Actual behavior When trying to build the debug image for the executor using the
executor:v1.8.1-debugimage to run the build, the build fails.~When trying to build an example image that leverages
RUN --mount=from=busybox,dst=/usr/as demonstrated in deploy/Dockerfile_debug the build works fine withdocker buildbut fails similarly to trying to build the debug image when using Kaniko.~Edit:
When trying to build an example image
FROM scratch, I have demonstrated in the comments that the issue is caused byexecutormounting the root of the container's filesystem in the container being built at each stage. Normally this behavior is not observed, because the mounting doesn't leave files laying around, however when the Dockerfile tries to copy files to /busybox or /kaniko and you use executor to build the image, executor fails to build the image.Expected behavior The build should complete. Building with
docker buildworks fine.To Reproduce Steps to reproduce the behavior:
docker build --build-arg TARGETARCH="amd64" -f deploy/Dockerfile_debug -t test:1 .docker run --rm -it --entrypoint "" -v "$(pwd)":/workspace test:1 /bin/shIn this example, the /usr directory doesn't appear to be being mounted. I noticed that
Dockerfile_debugsetsPATHto/usr/local/bin:/kaniko:/busyboxso I tried locally to change thedstin theRUNcommand to/usr/local/but still got the same error. So then I tried to change the["busybox", "sh", "-c", "mkdir -p /kaniko && chmod 777 /kaniko"]to use full paths:["/usr/bin/busybox", "sh", "-c", "mkdir -p /kaniko && chmod 777 /kaniko"]["/usr/local/bin/busybox", "sh", "-c", "mkdir -p /kaniko && chmod 777 /kaniko"]["/bin/busybox", "sh", "-c", "mkdir -p /kaniko && chmod 777 /kaniko"]["/busybox", "sh", "-c", "mkdir -p /kaniko && chmod 777 /kaniko"]Finally, I found that this works but then fails trying to run mkdir because the mkdir binary is not in the path either:
["/busybox/busybox", "sh", "-c", "mkdir -p /kaniko && chmod 777 /kaniko"]Additional Information
Dockerfile_debugfile from this repoKaniko Image (fully qualified with digest) Tried 2:
Triage Notes for the Maintainers
I came across some notes on docker builds related building from scratch here -- so I tried using
SHELLbefore theRUNcommand as described. That fixes this issue with not being able to find busybox in the path, but Kaniko should not require it, and it should work the same asdocker build. Additionally, once theSHELLwas added, the build failed immediately after starting the busybox shell when trying to runmkdiritself, becausemkdiris not in thePATHeither, same as what I described in the above section.So I made a test Dockerfile based on the first example from the link above, and I tried to build that with the same method used above for the debug container attempt. This also fails similar to my attempt to build the debug container without
SHELL:Example Dockerfile:
Kaniko build output:
However, when building with Docker, the build is successful.
I also verified that using the full path for the busybox binary fails:
Please advise if I'm doing anything wrong. It seems like /bin from the busybox image should be mounted in /usr/bin and working properly using the full path, so I think it's probably properly handling the
RUN --mount=fromflag properly but the overmounting by executor is placing a filesystem over / and therefore masking the mounted filesystem at /usr.--cacheflag