failed to get filesystem from image: error removing lib to make way for new symlink: unlinkat

pschichtel commented 3 years ago

Actual behavior Build of simple image with kaniko/executor:debug in docker with sysbox-runc runtime fails with:

error building image: error building stage: failed to get filesystem from image: error removing lib to make way for new symlink: unlinkat //lib/modules/5.10.0-8-amd64/build: read-only file system

Expected behavior

image builds just fine

To Reproduce Steps to reproduce the behavior:

install standard docker (20.10.8)
install sysbox-ce (0.4)
docker run --rm -it --entrypoint sh gcr.io/kaniko-project/executor:debug
create Dockerfile and packages
try building with /kaniko/executor --context . --dockerfile Dockerfile --destination "docker.repo.lbaum.eu/test123:latest" --cache=true --force

Additional Information

Dockerfile

FROM fedora:34
RUN dnf clean all
ADD packages /tmp/packages
RUN dnf --assumeyes install $(cat /tmp/packages)

Build Context packages:

autoconf
automake
binutils
bzip2
clang
curl
gettext
libcurl
libcurl-devel
libtool
libxml2
libxml2-devel
llvm
llvm-devel
make
maven
mingw-binutils-generic
mingw-filesystem-base
mingw-w64-tools
mingw32-binutils
mingw32-cpp
mingw32-crt
mingw32-curl-static
mingw32-expat-static
mingw32-filesystem
mingw32-gcc
mingw32-gcc-c++
mingw32-headers
mingw32-libidn-static
mingw32-libidn2-static
mingw32-libssh2-static
mingw32-nsis
mingw32-openssl-static
mingw32-pkg-config
mingw32-win-iconv-static
mingw32-winpthreads-static
mingw32-zlib-static
openssl
openssl-devel
patch
upx
uuid
uuid-devel
vim-common
wget
xar
xar-devel
xmlstarlet

Kaniko Image (fully qualified with digest) gcr.io/kaniko-project/executor@sha256:fcccd2ab9f3892e33fc7f2e950c8e4fc665e7a4c66f6a9d70b300d7a2103592f

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [x]
Please check if this error is seen when you use `--cache` flag	- [x]
Please check if your dockerfile is a multistage dockerfile	- [ ]

pschichtel commented 3 years ago

podman also works just fine:

podman --storage-driver=vfs build -t test .

pschichtel commented 3 years ago

this seems to fail when a base image has a "/lib" folder.

so far it happened for the library/fedora images and the jboss/keycloak (based on redhat's ubi8) image.

pschichtel commented 3 years ago

A workaround: --ignore-path=/lib

probably doesn't have an impact, but not entirely sure

pschichtel commented 3 years ago

It seems that --force is causing this (necessary with the latest :debug tag since the fix for container detection is not released yet).

pschichtel commented 3 years ago

On master this particular issue still happens and also a SIGSEGV is triggered when using the --ignore-path=/lib workaround:

# /kaniko/executor --context . --no-push --force --ignore-path=/lib
INFO[0000] Retrieving image manifest fedora:34          
INFO[0000] Retrieving image fedora:34 from registry index.docker.io 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest fedora:34          
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Unpacking rootfs as cmd RUN findmnt requires it. 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x7ff5e7]

goroutine 1 [running]:
github.com/GoogleContainerTools/kaniko/pkg/util.mkdirAllWithPermissions(0xc0000530c0, 0xf, 0x800001fd, 0x0, 0xc, 0x1, 0x1)
    /home/phillip/Code/kaniko/pkg/util/fs_util.go:776 +0x467
github.com/GoogleContainerTools/kaniko/pkg/util.ExtractFile(0x21d6f5c, 0x1, 0xc000e55a40, 0x2204180, 0xc0005f66c0, 0x0, 0x0)
    /home/phillip/Code/kaniko/pkg/util/fs_util.go:344 +0xe3c
github.com/GoogleContainerTools/kaniko/pkg/util.GetFSFromLayers(0x21d6f5c, 0x1, 0xc00062d2e0, 0x1, 0x1, 0xc00071eee0, 0x1, 0x1, 0x0, 0x0, ...)
    /home/phillip/Code/kaniko/pkg/util/fs_util.go:205 +0x78d
github.com/GoogleContainerTools/kaniko/pkg/util.GetFSFromImage(0x21d6f5c, 0x1, 0x2265948, 0xc00041bb20, 0x20056d0, 0xc000238c00, 0x0, 0xc00071ef60, 0x479525, 0x0)
    /home/phillip/Code/kaniko/pkg/util/fs_util.go:131 +0x147
github.com/GoogleContainerTools/kaniko/pkg/executor.(*stageBuilder).build.func1(0xc00071ef80, 0x47915c)
    /home/phillip/Code/kaniko/pkg/executor/build.go:311 +0x65
github.com/GoogleContainerTools/kaniko/pkg/util.Retry(0xc000e07278, 0x0, 0x3e8, 0x2f85d60, 0xc00071f288)
    /home/phillip/Code/kaniko/pkg/util/util.go:165 +0x2e
github.com/GoogleContainerTools/kaniko/pkg/executor.(*stageBuilder).build(0xc0004ad1e0, 0x0, 0x0)
    /home/phillip/Code/kaniko/pkg/executor/build.go:315 +0x147d
github.com/GoogleContainerTools/kaniko/pkg/executor.DoBuild(0x2f54860, 0x1, 0x0, 0x0, 0x1c86360)
    /home/phillip/Code/kaniko/pkg/executor/build.go:600 +0x645
github.com/GoogleContainerTools/kaniko/cmd/executor/cmd.glob..func2(0x2f3b8c0, 0xc000178000, 0x0, 0x5)
    /home/phillip/Code/kaniko/cmd/executor/cmd/root.go:141 +0xef
github.com/spf13/cobra.(*Command).execute(0x2f3b8c0, 0xc00004c1f0, 0x5, 0x5, 0x2f3b8c0, 0xc00004c1f0)
    /home/phillip/Code/kaniko/vendor/github.com/spf13/cobra/command.go:846 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0x2f3b8c0, 0xd, 0xc000798a50, 0x0)
    /home/phillip/Code/kaniko/vendor/github.com/spf13/cobra/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
    /home/phillip/Code/kaniko/vendor/github.com/spf13/cobra/command.go:887
main.main()
    /home/phillip/Code/kaniko/cmd/executor/main.go:31 +0x72

fuomag9 commented 2 years ago

same error even without --force

error building image: error building stage: failed to get filesystem from image: error removing lib to make way for new symlink: unlinkat //lib/modules/5.4.0-104-generic/initrd: read-only file system

aaron-prindle commented 1 year ago

possible dupe of https://github.com/GoogleContainerTools/kaniko/issues/1619

GoogleContainerTools / kaniko

failed to get filesystem from image: error removing lib to make way for new symlink: unlinkat #1745