Multi stage build hangs, continues to consume disk

[dlaidlaw]

Actual behavior Executing a multi stage build works with the docker command, but hangs using kaniko. The kaniko command runs for a very long time (more than 30 minutes). The last log output from kaniko is INFO[0058] Saving file . for later use

The same failure is observed running the kaniko build in either a gitlab-runner or from the Mac OSX command line. When run in the gitlab-runner (running in kubernetes) the build runs for about 8.5 minutes with an error message of ERROR: Job failed: command terminated with exit code 137

Expected behavior Kaniko should build the image. It should not take more than about 2 minutes. Building with the docker command using the same context and Dockerfile works OK, and takes between 15 and 30 seconds to run.

To Reproduce Steps to reproduce the behavior:

1. Execute kaniko using the gcr.io/kaniko-project/executor:debug with the Dockerfile and context below
2. Command:

   docker run -ti --rm --name build \
   --entrypoint "" \
   -v "$HOME/.aws:/root/.aws:ro" \
   -v "$(pwd)/build:/build" \
   -e "IMAGE=701234567890.dkr.ecr.us-east-1.amazonaws.com/gitlab-runner" \
   -e "TAG=v13.11.0" \
   -e "AWS_PROFILE=main" \
   -e "AWS_REGION=us-east-1" \
   -e "AWS_DEFAULT_REGION=us-east-1" \
   --workdir /build \
   gcr.io/kaniko-project/executor:debug \
   sh /build/build.sh

Additional Information

• Dockerfile

#
# Create a copy of the gitlab-runner with any local fixes.
#
# Changes:
# 1. Make the permissions on the entrypoint script 755
#
# Set baseimage and tag
ARG BASE=gitlab/gitlab-runner:v13.11.0

FROM gitlab/gitlab-runner:v13.11.0 AS baseimage

# Create app image
# Together with --squash parameter during build, it will squash even baseimage layers
FROM scratch
COPY --from=baseimage / /

# Fix World-Writable-Files from baseimage (must be here, it doesn't work when fixed in builder layer)
RUN chmod 755 /entrypoint 

# Put back the original gitlab-runner config

ENV PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV DEBIAN_FRONTEND=noninteractive
STOPSIGNAL SIGQUIT
CMD [ "run", "--user=gitlab-runner", "--working-directory=/home/gitlab-runner" ]
VOLUME [ "/etc/gitlab-runner", "/home/gitlab-runner" ]
ENTRYPOINT ["/usr/bin/dumb-init", "/entrypoint"]

• Build Context
  • Only one file, the command run to build the Dockerfile with kaniko

#!/usr/bin/env sh
#
# Copy the gitlab-runner, remove world writable files
#
echo '{"credsStore":"ecr-login"}' > /kaniko/.docker/config.json
/kaniko/executor --verbosity debug --context /build --dockerfile /build/Dockerfile --destination $IMAGE:$TAG  --destination $IMAGE:latest

• Kaniko Image (fully qualified with digest)

  • Digest: sha256:fcccd2ab9f3892e33fc7f2e950c8e4fc665e7a4c66f6a9d70b300d7a2103592f

• Log output from kaniko

DEBU[0000] Copying file /build/Dockerfile to /kaniko/Dockerfile
DEBU[0000] Skip resolving path /kaniko/Dockerfile
DEBU[0000] Skip resolving path /build
DEBU[0000] Skip resolving path /cache
DEBU[0000] Skip resolving path
DEBU[0000] Skip resolving path
DEBU[0000] Skip resolving path
DEBU[0000] Skip resolving path
INFO[0000] Resolved base name gitlab/gitlab-runner:v13.11.0 to baseimage
DEBU[0000] Built stage name to index map: map[baseimage:0]
INFO[0000] Retrieving image manifest gitlab/gitlab-runner:v13.11.0
INFO[0000] Retrieving image gitlab/gitlab-runner:v13.11.0 from registry index.docker.io
INFO[0001] No base image, nothing to extract
DEBU[0001] Resolved / to /
DEBU[0001] Resolved / to /
DEBU[0001] Replacing environment variable {PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin} with {PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin} in config
INFO[0001] Built cross stage deps: map[0:[/]]
INFO[0001] Retrieving image manifest gitlab/gitlab-runner:v13.11.0
INFO[0001] Returning cached image manifest
INFO[0001] Executing 0 build triggers
DEBU[0001] Mounted directories: [{/kaniko false} {/etc/mtab false} {/tmp/apt-key-gpghome true} {/var/run false} {/proc false} {/dev false} {/dev/pts false} {/sys false} {/sys/fs/cgroup false} {/sys/fs/cgroup/cpuset false} {/sys/fs/cgroup/cpu false} {/sys/fs/cgroup/cpuacct false} {/sys/fs/cgroup/blkio false} {/sys/fs/cgroup/memory false} {/sys/fs/cgroup/devices false} {/sys/fs/cgroup/freezer false} {/sys/fs/cgroup/net_cls false} {/sys/fs/cgroup/perf_event false} {/sys/fs/cgroup/net_prio false} {/sys/fs/cgroup/hugetlb false} {/sys/fs/cgroup/pids false} {/sys/fs/cgroup/rdma false} {/sys/fs/cgroup/systemd false} {/dev/mqueue false} {/dev/shm false} {/busybox false} {/build false} {/root/.aws false} {/etc/resolv.conf false} {/etc/hostname false} {/etc/hosts false} {/dev/console false} {/proc/bus false} {/proc/fs false} {/proc/irq false} {/proc/sys false} {/proc/sysrq-trigger false} {/proc/acpi false} {/proc/kcore false} {/proc/keys false} {/proc/timer_list false} {/proc/sched_debug false} {/sys/firmware false}]
DEBU[0001] Not adding /dev because it is ignored
DEBU[0001] Not adding /etc/hostname because it is ignored
DEBU[0001] Not adding /etc/hosts because it is ignored
DEBU[0001] Not adding /etc/resolv.conf because it is ignored
DEBU[0001] Not adding /proc because it is ignored
DEBU[0001] Not adding /sys because it is ignored
DEBU[0004] Not adding /var/run because it is ignored
DEBU[0004] Whiting out /run/systemd/.wh..wh..opq
DEBU[0004] not including whiteout files
DEBU[0045] Whiting out /tmp/.wh.gitlab-runner_amd64.deb
DEBU[0045] not including whiteout files
DEBU[0045] Whiting out /tmp/.wh.install-gitlab-runner
DEBU[0045] not including whiteout files
DEBU[0059] mapping stage idx 0 to digest sha256:bdc7dd7e64a46c9e86a5f6e1c9bc7b21c7aa5cbe6d05defe8d03d4eecc648193
DEBU[0059] mapping digest sha256:bdc7dd7e64a46c9e86a5f6e1c9bc7b21c7aa5cbe6d05defe8d03d4eecc648193 to cachekey
INFO[0059] Saving file . for later use

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [X]
Please check if this error is seen when you use --cache flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [X]

[f0o]

Same here :/

This only happens if you copy / to / - Seems to be looping somewhere and constantly eating up disk...

[dlaidlaw]

Yes, it seems to be having a problem copying the /dev directory from the source image. Still a bug, I think. Anyone have a workaround they use?

[aaron-prindle]

Possible dupe of https://github.com/GoogleContainerTools/kaniko/issues/1596