search

GoogleContainerTools / kaniko

Build Container Images In Kubernetes
Apache License 2.0
14.93k stars 1.45k forks source link

Kaniko is stucking on copying root #960

Open kvaps opened 4 years ago

kvaps commented 4 years ago

Actual behavior Kaniko is sucking forever when trying to save / (root) of previous stage

INFO[0007] Taking snapshot of full filesystem...        
INFO[0007] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0007] Saving file / for later use

Expected behavior Kaniko will copy / to the specified directory.

To Reproduce

mkdir -p /tmp/kaniko-bug
cd /tmp/kaniko-bug/

cat > Dockerfile <<\EOT
FROM alpine:3.11 as rootfs
RUN echo 7777

FROM alpine:3.11
COPY --from=rootfs / /sysroot/
EOT

docker run -ti --rm -v $PWD:/workspace gcr.io/kaniko-project/executor:v0.15.0 --dockerfile=Dockerfile --no-push

Additional Information

kvaps commented 4 years ago

This is my ugly workaround for this:

FROM alpine:3.11 as rootfs
RUN echo 7777

# Workaround https://github.com/GoogleContainerTools/kaniko/issues/960
RUN ROOTDIRS=$(find / -maxdepth 1 -mindepth 1 \( -type d -o -type l \)  ! -name builds ! -name busybox ! -name dev ! -name etc ! -name kaniko ! -name proc ! -name sys ! -name tmp ! -name var ! -name workspace) \
 && mkdir -p /rootfs/dev /rootfs/proc /rootfs/run /rootfs/sys /rootfs/tmp \
 && cp -ax /etc/ /var /rootfs \
 && rm -rf /rootfs/var/run \
 && ln -s ../run/ /rootfs/var/run \
 && mv $ROOTDIRS /rootfs/

FROM alpine:3.11
COPY --from=rootfs /rootfs/ /sysroot/
kvaps commented 4 years ago

/area multi-stage builds /kind bug

cvgw commented 4 years ago

Unfortunately I think this behavior is expected. There are directories at / (such as /kaniko) that are "special"/"reserved". I'm not sure if there is a better work around than you've suggested

kvaps commented 4 years ago

But docker and buildkit is working fine with this

cvgw commented 4 years ago

But docker and buildkit is working fine with this

Right, this is specific to the way that kaniko is implemented.

invokermain commented 2 years ago

For what its worth I've run into this when running a COPY command in my Dockerfile using environment variables that don't exist.

e.g. COPY --from=builder $PYSETUP_PATH $PYSETUP_PATH will hang on Saving file . for later use if $PYSETUP_PATH is not defined/default. I guess it might be trying to do COPY --from=builder . . which obviously doesn't make sense.

pmhahn commented 2 years ago

I'm using debootstrap to build build a base image using the following Dockerfile:

FROM debian:bullseye-slim AS builder
RUN apt-get -qq update && apt-get -q install --assume-yes debootstrap findutils
RUN debootstrap --no-merged-usr --variant='minbase'  stable /work http://deb.debian.org/
FROM scratch
COPY --from=builder /work /

which stalls when kaniko copies the content of /dev/console instead of handling it as a special file. The same Dockerfile works fine with docker. I have created https://github.com/otiai10/copy/issues/78 to implement handling special files with otiai10/copy which is used by kaniko for copying.

pmhahn commented 2 years ago

Following the hint from otiai10/copy#78 Skip could be used to at least not copy the content, e.g. something like this:

opt := Options{
    Skip: func(src string) (bool, error) {
        stat, err := os.Stat(src)
        if err != nil {
            return nil, err
        }
        return stat.mode & (os.ModeDevice | os.ModeNamedPipe | os.ModeSocket) == 0, nil
    },
}
err := Copy("your/directory", "your/directory.copy", opt)

PS: I'm no Go programmer, so Syntax may be wrong.

aaron-prindle commented 1 year ago

It seems a fix was added to otiai related to this specific issue where it's defaults were changed + functionality added to handle special files, PR here https://github.com/otiai10/copy/pull/84

When I attempt the repro Dockerfile suggested above though, I am still seeing a Kaniko build failure despite using an otiai version with that fix PR:

INFO[0010] Pushing image to gcr.io/aprindle-test-cluster/kaniko-test/cache:900ada9315de8b51c19436ce83cf56ade4e49ffb0d88ad4385093856925b5423 
I: Target architecture can be executed
I: Retrieving InRelease 
I: Retrieving Release 
E: Failed getting release file http://deb.debian.org/dists/stable/Release
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1

keeping this open for now

lc-guy commented 1 year ago

Still hitting this issue.

Considering kaniko doesn't support the --squash command-line argument to reduce an image to a single layer, copying the entire rootfs is the only way to achieve that goal, and this bug makes it wholly impossible, sadly.

My use case is that I'm stripping down an existing very large image to remove cruft I don't need, but of course it'll just stack on more layers as you remove the files, so flattening the image is needed afterwards.

PylotLight commented 1 month ago

I tried using .dockerignore workaround to include the kaniko dir and the newer exclude option which both have not worked. COPY --exclude=kaniko --from=xx / /

Any other potential updates on this one causing the bug:

error building image: could not save file: copying file: open /kaniko/0/kaniko/0..../kaniko/.docker/..2024_10_21_02_14_45.4005423025/config.json: file name too long