Closed tanguydelignieresaccenture closed 4 years ago
cvgw
commented
4 years ago I'm able to repro (thank you @tanguydelignieresaccenture for the detailed report)
I'm seeing some strange output in a debugger at util/fs_util.go:158
print path => "/root/buildinfo/.wh..wh..opq"
print path => "/run/secrets/.wh..wh..opq"It looks like files are being double whiteouted? I can't image a scenario where we would expect a file to have double .wh. prefixes like the examples above.
cvgw
commented
4 years ago at util/fs_util.go
if err := os.RemoveAllI(filiepath.Join(dir, name)); err != nil {
dir => /
name => dev
path => /.wh.dev
os.RemoveAll("/dev") // OH NO!ya trying to remove /dev is gonna be a bad time
cvgw
commented
4 years ago /dev is listed in the whitelists (I'm not sure why, is it a volume?)
{Path: "/dev", PrefixMatchOnly: false}If /dev is whitelisted then I would assume it doesn't get whiteout, but it seems that it does.
cvgw
commented
4 years ago Looks like /dev gets whitelisted as part of this /proc logic (that I didn't know we had)
> github.com/GoogleContainerTools/kaniko/pkg/util.DetectFilesystemWhitelist() .go/src/github.com/GoogleContainerTools/kaniko/pkg/util/fs_util.go:407 (PC: 0x17128df)
402: }
403: defer f.Close()
404: reader := bufio.NewReader(f)
405: for {
406: line, err := reader.ReadString('\n')
=> 407: logrus.Tracef("Read the following line from %s: %s", path, line)
408: if err != nil && err != io.EOF {
409: return err
410: }
411: lineArr := strings.Split(line, " ")
412: if len(lineArr) < 5 {
(dlv) print line
"951 949 0:82 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=...+4 more"I have a theory about this issue
In https://github.com/GoogleContainerTools/kaniko/pull/971 this code was added https://github.com/GoogleContainerTools/kaniko/blob/8d9e6b8ea54274f73517f11c113c13cd03d26349/pkg/snapshot/snapshot.go#L183
In the issue example, this causes /etc/systemd/system/systemd-logind.service and its resolved link /dev/null to be appended to the filesToAdd list. The issue with this is that /dev is whitelisted and all of it's subdirectories should be skipped.
I think the solution here would be moving the filesWithLinks logic to the Walk block
https://github.com/GoogleContainerTools/kaniko/blob/8d9e6b8ea54274f73517f11c113c13cd03d26349/pkg/snapshot/snapshot.go#L133
or duplicating the whitelist checks in filesWithLinks
@tejal29 @samos123 WDYT?
cvgw
commented
4 years ago @tanguydelignieresaccenture can you try tag 1039-fix-test or debug-1039-fix-test?
I've just pushed these two tags and I'd like to test if they fix the issue you are seeing. Thanks!
RemcodM
commented
4 years ago @cvgw After your observations that the issues #1038 #1024 and #1028 might all be related, and my involvement with the issues #1024 and #1028, I figured I could do a test on my own.
I used the following Docker image:
# Eoan amd64 image
FROM ubuntu:eoan
ADD build /usr/local/bin/
ADD backports /backports
RUN apt-get update && \
export DEBIAN_FRONTEND=noninteractive && \
apt-get dist-upgrade -y && \
apt-get install -qy sudo build-essential devscripts equivs lsb-release apt-transport-https && \
cd /backports && \
apt-get install -qy ./libdebhelper-perl_12.8_all.deb ./debhelper_12.8_all.deb ./dh-systemd_12.8_all.deb && \
apt-get clean -y && \
useradd -c "Build User" -d /tmp -M -s /bin/bash builduserI have build this Docker image 4 times: One time using tag debug-v0.16.0, one time using debug-v0.17.0, one time using debug-v0.17.1 and one last time using debug-1039-fix-test. I have build this exact Dockerfile succesfully before on debug-v0.16.0 and I have seen builds of this file gone wrong on debug-v0.17.0 before.
These are the results:
# docker pull registry.tworem.nl/remcodm/kaniko-test/eoan/amd64:v0.16.0
v0.16.0: Pulling from remcodm/kaniko-test/eoan/amd64
93956c6f8d9e: Already exists
46bddb84d1c5: Already exists
15fa85048576: Already exists
8aa40341c4fa: Already exists
08d396ded471: Pull complete
4ca40aa44069: Pull complete
96ef31c5e575: Pull complete
Digest: sha256:7305979dfc0692645ebc72bd7e81c31bd2c974a457f7698cf1630b6e2441d65a
Status: Downloaded newer image for registry.tworem.nl/remcodm/kaniko-test/eoan/amd64:v0.16.0
registry.tworem.nl/remcodm/kaniko-test/eoan/amd64:v0.16.0# docker pull registry.tworem.nl/remcodm/kaniko-test/eoan/amd64:v0.17.0
v0.17.0: Pulling from remcodm/kaniko-test/eoan/amd64
93956c6f8d9e: Already exists
46bddb84d1c5: Already exists
15fa85048576: Already exists
8aa40341c4fa: Already exists
c72f51425616: Pull complete
e7ae8616318f: Pull complete
73ba141bedd1: Extracting [==================================================>] 177.3MB/177.3MB
failed to register layer: ApplyLayer exit status 1 stdout: stderr: link /lib/systemd /usr/lib/systemd: no such file or directory# docker pull registry.tworem.nl/remcodm/kaniko-test/eoan/amd64:v0.17.1
v0.17.1: Pulling from remcodm/kaniko-test/eoan/amd64
93956c6f8d9e: Already exists
46bddb84d1c5: Already exists
15fa85048576: Already exists
8aa40341c4fa: Already exists
97d612b5000c: Pull complete
6eaaf8e851c4: Pull complete
53c75cb1d7a9: Extracting [==================================================>] 177.3MB/177.3MB
failed to register layer: ApplyLayer exit status 1 stdout: stderr: link /lib/systemd /usr/lib/systemd: no such file or directory# docker pull registry.tworem.nl/remcodm/kaniko-test/eoan/amd64:1039-fix-test
1039-fix-test: Pulling from remcodm/kaniko-test/eoan/amd64
93956c6f8d9e: Already exists
46bddb84d1c5: Already exists
15fa85048576: Already exists
8aa40341c4fa: Already exists
cf5e7c37f679: Pull complete
e4b591f8376c: Pull complete
31603ad68aef: Extracting [==================================================>] 177.3MB/177.3MB
failed to register layer: ApplyLayer exit status 1 stdout: stderr: link /lib/systemd /usr/lib/systemd: no such file or directorySo for me, debug-1039-fix-test did not resolve this issue I'm seeing. As you can see, debug-v0.16.0 does work fine.
RemcodM
commented
4 years ago One more comment: I am not sure if it is related or useful at all, but I noticed when going over the build logs that debug-v0.16.0 seems to add some whiteouts for files at the end of the last layer, while version debug-v0.17.0 and later do not. Same for debug-1039-fix-test.
Build log debug-v0.16.0: https://git.tworem.nl/RemcodM/kaniko-test/-/jobs/17363/raw
Build log debug-v0.17.0: https://git.tworem.nl/RemcodM/kaniko-test/-/jobs/17364/raw
Build log debug-v0.17.1: https://git.tworem.nl/RemcodM/kaniko-test/-/jobs/17365/raw
Build log debug-1039-fix-test: https://git.tworem.nl/RemcodM/kaniko-test/-/jobs/17366/raw
tanguydelignieresaccenture
commented
4 years ago @cvgw, I've made a test with debug-1039-fix-test tag, following the same steps I gave in this issue, and it's OK for me.
@RemcodM maybe you're issue is not the same, as I did not get pull error in debug-v0.17.1 nor debug-1039-fix-test.
Here are the results:
gcr.io/kaniko-project/executor:debug-1039-fix-test
$ docker run --rm --entrypoint "" -v /host/path/to/kaniko/config.json:/kaniko/.docker/config.json -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build1:/workspace gcr.io/kaniko-project/executor:debug-1039-fix-test /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile --destination index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test
INFO[0002] Resolved base name registry.access.redhat.com/ubi7/ubi:7.7-214 to registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0002] Resolved base name registry.access.redhat.com/ubi7/ubi:7.7-214 to registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0002] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0004] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0005] Built cross stage deps: map[]
INFO[0005] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0006] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0008] Unpacking rootfs as cmd RUN yum --disableplugin=subscription-manager update -y && yum --disableplugin=subscription-manager install -y gcc-4.8.5-39.el7 gcc-c++-4.8.5-39.el7 make-3.82-24.el7 && yum --disableplugin=subscription-manager clean all requires it.
INFO[0122] Taking snapshot of full filesystem...
INFO[0124] RUN yum --disableplugin=subscription-manager update -y && yum --disableplugin=subscription-manager install -y gcc-4.8.5-39.el7 gcc-c++-4.8.5-39.el7 make-3.82-24.el7 && yum --disableplugin=subscription-manager clean all
INFO[0124] cmd: /bin/sh
INFO[0124] args: [-c yum --disableplugin=subscription-manager update -y && yum --disableplugin=subscription-manager install -y gcc-4.8.5-39.el7 gcc-c++-4.8.5-39.el7 make-3.82-24.el7 && yum --disableplugin=subscription-manager clean all]
Loaded plugins: ovl, product-id, search-disabled-repos
https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi/server/7/7Server/x86_64/devtools/1/os/repodata/a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml: [Errno 12] Timeout on https://cdn-ubi.redhat.com/content/public/ubi/dist/ubi/server/7/7Server/x86_64/devtools/1/os/repodata/a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml: (28, 'Operation timed out after 30001 milliseconds with 0 out of 0 bytes received')
Trying other mirror.
Resolving Dependencies
--> Running transaction check
---> Package binutils.x86_64 0:2.27-41.base.el7_7.1 will be updated
---> Package binutils.x86_64 0:2.27-41.base.el7_7.2 will be an update
---> Package curl.x86_64 0:7.29.0-54.el7_7.1 will be updated
---> Package curl.x86_64 0:7.29.0-54.el7_7.2 will be an update
---> Package libcurl.x86_64 0:7.29.0-54.el7_7.1 will be updated
---> Package libcurl.x86_64 0:7.29.0-54.el7_7.2 will be an update
---> Package nss.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-softokn.x86_64 0:3.44.0-5.el7 will be updated
---> Package nss-softokn.x86_64 0:3.44.0-8.el7_7 will be an update
---> Package nss-softokn-freebl.x86_64 0:3.44.0-5.el7 will be updated
---> Package nss-softokn-freebl.x86_64 0:3.44.0-8.el7_7 will be an update
---> Package nss-sysinit.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss-sysinit.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-tools.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss-tools.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-util.x86_64 0:3.44.0-3.el7 will be updated
---> Package nss-util.x86_64 0:3.44.0-4.el7_7 will be an update
---> Package sqlite.x86_64 0:3.7.17-8.el7 will be updated
---> Package sqlite.x86_64 0:3.7.17-8.el7_7.1 will be an update
---> Package systemd.x86_64 0:219-67.el7_7.2 will be updated
---> Package systemd.x86_64 0:219-67.el7_7.3 will be an update
---> Package systemd-libs.x86_64 0:219-67.el7_7.2 will be updated
---> Package systemd-libs.x86_64 0:219-67.el7_7.3 will be an update
--> Finished Dependency ResolutionDependencies Resolved
Updating: binutils x86_64 2.27-41.base.el7_7.2 ubi-7 5.9 M curl x86_64 7.29.0-54.el7_7.2 ubi-7 270 k libcurl x86_64 7.29.0-54.el7_7.2 ubi-7 223 k nss x86_64 3.44.0-7.el7_7 ubi-7 854 k nss-softokn x86_64 3.44.0-8.el7_7 ubi-7 330 k nss-softokn-freebl x86_64 3.44.0-8.el7_7 ubi-7 224 k nss-sysinit x86_64 3.44.0-7.el7_7 ubi-7 65 k nss-tools x86_64 3.44.0-7.el7_7 ubi-7 528 k nss-util x86_64 3.44.0-4.el7_7 ubi-7 79 k sqlite x86_64 3.7.17-8.el7_7.1 ubi-7 394 k systemd x86_64 219-67.el7_7.3 ubi-7 5.1 M systemd-libs x86_64 219-67.el7_7.3 ubi-7 411 k
Upgrade 12 Packages
Total 768 kB/s | 14 MB 00:19 Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release Importing GPG key 0xFD431D51: Userid : "Red Hat, Inc. (release key 2) security@redhat.com" Fingerprint: 567e 347a d004 4ade 55ba 8a5f 199e 2f91 fd43 1d51 Package : redhat-release-server-7.7-10.el7.x86_64 (@anaconda/7.7) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release Importing GPG key 0x2FA658E0: Userid : "Red Hat, Inc. (auxiliary key) security@redhat.com" Fingerprint: 43a6 e49c 4a38 f4be 9abf 2a53 4568 9c88 2fa6 58e0 Package : redhat-release-server-7.7-10.el7.x86_64 (@anaconda/7.7) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : nss-util-3.44.0-4.el7_7.x86_64 1/24 Updating : nss-softokn-freebl-3.44.0-8.el7_7.x86_64 2/24 Updating : systemd-libs-219-67.el7_7.3.x86_64 3/24 Updating : sqlite-3.7.17-8.el7_7.1.x86_64 4/24 Updating : nss-softokn-3.44.0-8.el7_7.x86_64 5/24 Updating : nss-sysinit-3.44.0-7.el7_7.x86_64 6/24 Updating : nss-3.44.0-7.el7_7.x86_64 7/24 Updating : libcurl-7.29.0-54.el7_7.2.x86_64 8/24 Updating : curl-7.29.0-54.el7_7.2.x86_64 9/24 Updating : systemd-219-67.el7_7.3.x86_64 10/24 Failed to get D-Bus connection: Operation not permitted Updating : nss-tools-3.44.0-7.el7_7.x86_64 11/24 Updating : binutils-2.27-41.base.el7_7.2.x86_64 12/24 install-info: No such file or directory for /usr/share/info/as.info.gz install-info: No such file or directory for /usr/share/info/binutils.info.gz install-info: No such file or directory for /usr/share/info/gprof.info.gz install-info: No such file or directory for /usr/share/info/ld.info.gz install-info: No such file or directory for /usr/share/info/standards.info.gz Cleanup : nss-tools-3.44.0-4.el7.x86_64 13/24 Cleanup : curl-7.29.0-54.el7_7.1.x86_64 14/24 Cleanup : systemd-219-67.el7_7.2.x86_64 15/24 Cleanup : libcurl-7.29.0-54.el7_7.1.x86_64 16/24 Cleanup : nss-sysinit-3.44.0-4.el7.x86_64 17/24 Cleanup : nss-3.44.0-4.el7.x86_64 18/24 Cleanup : nss-softokn-3.44.0-5.el7.x86_64 19/24 Cleanup : nss-softokn-freebl-3.44.0-5.el7.x86_64 20/24 Cleanup : nss-util-3.44.0-3.el7.x86_64 21/24 Cleanup : sqlite-3.7.17-8.el7.x86_64 22/24 Cleanup : systemd-libs-219-67.el7_7.2.x86_64 23/24 Cleanup : binutils-2.27-41.base.el7_7.1.x86_64 24/24 Verifying : binutils-2.27-41.base.el7_7.2.x86_64 1/24 Verifying : nss-tools-3.44.0-7.el7_7.x86_64 2/24 Verifying : nss-3.44.0-7.el7_7.x86_64 3/24 Verifying : sqlite-3.7.17-8.el7_7.1.x86_64 4/24 Verifying : systemd-libs-219-67.el7_7.3.x86_64 5/24 Verifying : nss-sysinit-3.44.0-7.el7_7.x86_64 6/24 Verifying : curl-7.29.0-54.el7_7.2.x86_64 7/24 Verifying : systemd-219-67.el7_7.3.x86_64 8/24 Verifying : libcurl-7.29.0-54.el7_7.2.x86_64 9/24 Verifying : nss-util-3.44.0-4.el7_7.x86_64 10/24 Verifying : nss-softokn-freebl-3.44.0-8.el7_7.x86_64 11/24 Verifying : nss-softokn-3.44.0-8.el7_7.x86_64 12/24 Verifying : nss-tools-3.44.0-4.el7.x86_64 13/24 Verifying : systemd-libs-219-67.el7_7.2.x86_64 14/24 Verifying : systemd-219-67.el7_7.2.x86_64 15/24 Verifying : nss-util-3.44.0-3.el7.x86_64 16/24 Verifying : curl-7.29.0-54.el7_7.1.x86_64 17/24 Verifying : libcurl-7.29.0-54.el7_7.1.x86_64 18/24 Verifying : nss-sysinit-3.44.0-4.el7.x86_64 19/24 Verifying : binutils-2.27-41.base.el7_7.1.x86_64 20/24 Verifying : nss-softokn-freebl-3.44.0-5.el7.x86_64 21/24 Verifying : nss-3.44.0-4.el7.x86_64 22/24 Verifying : nss-softokn-3.44.0-5.el7.x86_64 23/24 Verifying : sqlite-3.7.17-8.el7.x86_64 24/24
Updated: binutils.x86_64 0:2.27-41.base.el7_7.2 curl.x86_64 0:7.29.0-54.el7_7.2 libcurl.x86_64 0:7.29.0-54.el7_7.2 nss.x86_64 0:3.44.0-7.el7_7 nss-softokn.x86_64 0:3.44.0-8.el7_7 nss-softokn-freebl.x86_64 0:3.44.0-8.el7_7 nss-sysinit.x86_64 0:3.44.0-7.el7_7 nss-tools.x86_64 0:3.44.0-7.el7_7 nss-util.x86_64 0:3.44.0-4.el7_7 sqlite.x86_64 0:3.7.17-8.el7_7.1 systemd.x86_64 0:219-67.el7_7.3 systemd-libs.x86_64 0:219-67.el7_7.3
Complete! Loaded plugins: ovl, product-id, search-disabled-repos Resolving Dependencies --> Running transaction check ---> Package gcc.x86_64 0:4.8.5-39.el7 will be installed --> Processing Dependency: cpp = 4.8.5-39.el7 for package: gcc-4.8.5-39.el7.x86_64 --> Processing Dependency: libgomp = 4.8.5-39.el7 for package: gcc-4.8.5-39.el7.x86_64 --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-4.8.5-39.el7.x86_64 --> Processing Dependency: libgomp.so.1()(64bit) for package: gcc-4.8.5-39.el7.x86_64 --> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-4.8.5-39.el7.x86_64 --> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-4.8.5-39.el7.x86_64 ---> Package gcc-c++.x86_64 0:4.8.5-39.el7 will be installed --> Processing Dependency: libstdc++-devel = 4.8.5-39.el7 for package: gcc-c++-4.8.5-39.el7.x86_64 ---> Package make.x86_64 1:3.82-24.el7 will be installed --> Running transaction check ---> Package cpp.x86_64 0:4.8.5-39.el7 will be installed ---> Package glibc-devel.x86_64 0:2.17-292.el7 will be installed --> Processing Dependency: glibc-headers = 2.17-292.el7 for package: glibc-devel-2.17-292.el7.x86_64 --> Processing Dependency: glibc-headers for package: glibc-devel-2.17-292.el7.x86_64 ---> Package libgomp.x86_64 0:4.8.5-39.el7 will be installed ---> Package libmpc.x86_64 0:1.0.1-3.el7 will be installed ---> Package libstdc++-devel.x86_64 0:4.8.5-39.el7 will be installed ---> Package mpfr.x86_64 0:3.1.1-4.el7 will be installed --> Running transaction check ---> Package glibc-headers.x86_64 0:2.17-292.el7 will be installed --> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.17-292.el7.x86_64 --> Processing Dependency: kernel-headers for package: glibc-headers-2.17-292.el7.x86_64 --> Running transaction check ---> Package kernel-headers.x86_64 0:3.10.0-1062.12.1.el7 will be installed --> Finished Dependency Resolution
Dependencies Resolved
Installing: gcc x86_64 4.8.5-39.el7 ubi-7 16 M gcc-c++ x86_64 4.8.5-39.el7 ubi-7 7.2 M make x86_64 1:3.82-24.el7 ubi-7 421 k Installing for dependencies: cpp x86_64 4.8.5-39.el7 ubi-7 6.0 M glibc-devel x86_64 2.17-292.el7 ubi-7 1.1 M glibc-headers x86_64 2.17-292.el7 ubi-7 687 k kernel-headers x86_64 3.10.0-1062.12.1.el7 ubi-7 8.7 M libgomp x86_64 4.8.5-39.el7 ubi-7 158 k libmpc x86_64 1.0.1-3.el7 ubi-7 51 k libstdc++-devel x86_64 4.8.5-39.el7 ubi-7 1.5 M mpfr x86_64 3.1.1-4.el7 ubi-7 203 k
Install 3 Packages (+8 Dependent packages)
Total 778 kB/s | 42 MB 00:55 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : mpfr-3.1.1-4.el7.x86_64 1/11 Installing : libmpc-1.0.1-3.el7.x86_64 2/11 Installing : cpp-4.8.5-39.el7.x86_64 3/11 Installing : kernel-headers-3.10.0-1062.12.1.el7.x86_64 4/11 Installing : glibc-headers-2.17-292.el7.x86_64 5/11 Installing : glibc-devel-2.17-292.el7.x86_64 6/11 Installing : libstdc++-devel-4.8.5-39.el7.x86_64 7/11 Installing : libgomp-4.8.5-39.el7.x86_64 8/11 Installing : gcc-4.8.5-39.el7.x86_64 9/11 Installing : gcc-c++-4.8.5-39.el7.x86_64 10/11 Installing : 1:make-3.82-24.el7.x86_64 11/11 Verifying : glibc-devel-2.17-292.el7.x86_64 1/11 Verifying : gcc-c++-4.8.5-39.el7.x86_64 2/11 Verifying : 1:make-3.82-24.el7.x86_64 3/11 Verifying : mpfr-3.1.1-4.el7.x86_64 4/11 Verifying : libgomp-4.8.5-39.el7.x86_64 5/11 Verifying : libmpc-1.0.1-3.el7.x86_64 6/11 Verifying : cpp-4.8.5-39.el7.x86_64 7/11 Verifying : gcc-4.8.5-39.el7.x86_64 8/11 Verifying : libstdc++-devel-4.8.5-39.el7.x86_64 9/11 Verifying : kernel-headers-3.10.0-1062.12.1.el7.x86_64 10/11 Verifying : glibc-headers-2.17-292.el7.x86_64 11/11
Installed: gcc.x86_64 0:4.8.5-39.el7 gcc-c++.x86_64 0:4.8.5-39.el7 make.x86_64 1:3.82-24.el7
Dependency Installed: cpp.x86_64 0:4.8.5-39.el7 glibc-devel.x86_64 0:2.17-292.el7 glibc-headers.x86_64 0:2.17-292.el7 kernel-headers.x86_64 0:3.10.0-1062.12.1.el7 libgomp.x86_64 0:4.8.5-39.el7 libmpc.x86_64 0:1.0.1-3.el7 libstdc++-devel.x86_64 0:4.8.5-39.el7 mpfr.x86_64 0:3.1.1-4.el7
Complete! Loaded plugins: ovl, product-id, search-disabled-repos Cleaning repos: ubi-7 ubi-7-rhah ubi-7-server-devtools-rpms : ubi-7-server-extras-rpms ubi-7-server-optional-rpms : ubi-server-rhscl-7-rpms INFO[0255] Taking snapshot of full filesystem...
2. Try to build the image from the Dockerfile "build2" with `gcr.io/kaniko-project/executor:debug-1039-fix-test`, using the image "build1" with tag `debug-1039-fix-test` (see `Dockerfile.debug-1039-fix-test`) > OK
```shell
$ docker run --rm --entrypoint "" -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build2:/workspace gcr.io/kaniko-project/executor:debug-1039-fix-test /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile.debug-1039-fix-test --no-push
INFO[0000] Resolved base name index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test to index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test
INFO[0000] Resolved base name index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test to index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test
INFO[0000] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test
INFO[0004] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test
INFO[0008] Built cross stage deps: map[]
INFO[0008] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test
INFO[0009] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test
INFO[0011] Unpacking rootfs as cmd RUN echo test requires it.
INFO[0178] Taking snapshot of full filesystem...
INFO[0179] RUN echo test
INFO[0179] cmd: /bin/sh
INFO[0179] args: [-c echo test]
test
INFO[0179] Taking snapshot of full filesystem...
INFO[0180] Skipping push to container registry due to --no-push flagDockerfile.debug-1039-fix-test
FROM index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test
RUN echo test
index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-1039-fix-test available on my dockerhub.
olivierboudet
commented
4 years ago @tanguydelignieresaccenture can you try tag
1039-fix-testordebug-1039-fix-test?I've just pushed these two tags and I'd like to test if they fix the issue you are seeing. Thanks!
I have tested this image on my side too and it fix my similar issue.
jlucktay
commented
4 years ago I just started encountering this issue this morning. My Dockerfile is here and the associated build script and Cloud Build YAML are alongside in the same directory. It has failed with the same error 4 times in a row just now.
dinvlad
commented
4 years ago I'm getting the same error even with debug-v0.16.0, for the situation described in #1033. So atm there doesn't appear to be a workaround (other than using regular Docker builds)..
kenanduman1988
commented
4 years ago I'm getting the same error as well with gcr.io/kaniko-project/executor:debug
diegosucaria
commented
4 years ago same error here!
tejal29
commented
4 years ago @dinvlad Can you mention your error?
debug-v-.16.0 should work if it was due to a regression.
dinvlad
commented
4 years ago @tejal29 the error is the same as in this issue, and the error reproduction and log are described in https://github.com/GoogleContainerTools/kaniko/issues/1033 (however, the workaround no longer seems to work, even with 0.16.0).
dinvlad
commented
4 years ago This is breaking all of our multi-stage builds currently (i.e. not limited to the situation described in #1033). Moreover, switching to v0.16.0 doesn’t help. The only workaround is to switch to normal Docker builds (i.e. not using Kaniko) :(
dinvlad
commented
4 years ago OK, apparently this issue has to do with caching. If I "break" the cache (either by removing all images from the cache directory on GCR, or by simply modifying a base layer) then the full rebuild succeeds. But when I rebuild it again (with the new cache), it fails..
conblem
commented
4 years ago @dinvlad same issue here
cvgw
commented
4 years ago We've committed a change which I hope will fix this. If anyone feels like testing tags a1af057f997316bfb1c4d2d82719d78481a02a79 and debug-a1af057f997316bfb1c4d2d82719d78481a02a79 have the new code
tanguydelignieresaccenture
commented
4 years ago I have re-run this issue context with the debug-a1af057f997316bfb1c4d2d82719d78481a02a79 tag and have confirm it fix it.
@cvgw, do you want details of test run?
Update: here it is
-- build 1
$ docker run --rm --entrypoint "" -v /host/path/to/kaniko/config.json:/kaniko/.docker/config.json -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build1:/workspace gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile --destination index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
Unable to find image 'gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79' locally
debug-a1af057f997316bfb1c4d2d82719d78481a02a79: Pulling from kaniko-project/executor
d746bf49e551: Pull complete 55bd16948852: Pull complete e6a8536b92d3: Pull complete ce4f88fff860: Pull complete 956cd6dc6b64: Pull complete 56587e5229ac: Pull complete 1ebb4d9355e1: Pull complete Digest: sha256:dd28123388927a87f01c75dc6fddc222fe79273139fa040f8b4fa3045bacf538
Status: Downloaded newer image for gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
INFO[0001] Resolved base name registry.access.redhat.com/ubi7/ubi:7.7-214 to registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0001] Resolved base name registry.access.redhat.com/ubi7/ubi:7.7-214 to registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0001] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0004] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0006] Built cross stage deps: map[]
INFO[0006] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0008] Retrieving image manifest registry.access.redhat.com/ubi7/ubi:7.7-214
INFO[0009] Unpacking rootfs as cmd RUN yum --disableplugin=subscription-manager update -y && yum --disableplugin=subscription-manager install -y gcc-4.8.5-39.el7 gcc-c++-4.8.5-39.el7 make-3.82-24.el7 && yum --disableplugin=subscription-manager clean all requires it.
INFO[0025] Taking snapshot of full filesystem...
INFO[0025] Resolving paths
INFO[0027] RUN yum --disableplugin=subscription-manager update -y && yum --disableplugin=subscription-manager install -y gcc-4.8.5-39.el7 gcc-c++-4.8.5-39.el7 make-3.82-24.el7 && yum --disableplugin=subscription-manager clean all
INFO[0027] cmd: /bin/sh
INFO[0027] args: [-c yum --disableplugin=subscription-manager update -y && yum --disableplugin=subscription-manager install -y gcc-4.8.5-39.el7 gcc-c++-4.8.5-39.el7 make-3.82-24.el7 && yum --disableplugin=subscription-manager clean all]
Loaded plugins: ovl, product-id, search-disabled-repos
Resolving Dependencies
--> Running transaction check
---> Package binutils.x86_64 0:2.27-41.base.el7_7.1 will be updated
---> Package binutils.x86_64 0:2.27-41.base.el7_7.2 will be an update
---> Package curl.x86_64 0:7.29.0-54.el7_7.1 will be updated
---> Package curl.x86_64 0:7.29.0-54.el7_7.2 will be an update
---> Package libcurl.x86_64 0:7.29.0-54.el7_7.1 will be updated
---> Package libcurl.x86_64 0:7.29.0-54.el7_7.2 will be an update
---> Package nss.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-softokn.x86_64 0:3.44.0-5.el7 will be updated
---> Package nss-softokn.x86_64 0:3.44.0-8.el7_7 will be an update
---> Package nss-softokn-freebl.x86_64 0:3.44.0-5.el7 will be updated
---> Package nss-softokn-freebl.x86_64 0:3.44.0-8.el7_7 will be an update
---> Package nss-sysinit.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss-sysinit.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-tools.x86_64 0:3.44.0-4.el7 will be updated
---> Package nss-tools.x86_64 0:3.44.0-7.el7_7 will be an update
---> Package nss-util.x86_64 0:3.44.0-3.el7 will be updated
---> Package nss-util.x86_64 0:3.44.0-4.el7_7 will be an update
---> Package sqlite.x86_64 0:3.7.17-8.el7 will be updated
---> Package sqlite.x86_64 0:3.7.17-8.el7_7.1 will be an update
---> Package systemd.x86_64 0:219-67.el7_7.2 will be updated
---> Package systemd.x86_64 0:219-67.el7_7.3 will be an update
---> Package systemd-libs.x86_64 0:219-67.el7_7.2 will be updated
---> Package systemd-libs.x86_64 0:219-67.el7_7.3 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Updating:
binutils x86_64 2.27-41.base.el7_7.2 ubi-7 5.9 M
curl x86_64 7.29.0-54.el7_7.2 ubi-7 270 k
libcurl x86_64 7.29.0-54.el7_7.2 ubi-7 223 k
nss x86_64 3.44.0-7.el7_7 ubi-7 854 k
nss-softokn x86_64 3.44.0-8.el7_7 ubi-7 330 k
nss-softokn-freebl x86_64 3.44.0-8.el7_7 ubi-7 224 k
nss-sysinit x86_64 3.44.0-7.el7_7 ubi-7 65 k
nss-tools x86_64 3.44.0-7.el7_7 ubi-7 528 k
nss-util x86_64 3.44.0-4.el7_7 ubi-7 79 k
sqlite x86_64 3.7.17-8.el7_7.1 ubi-7 394 k
systemd x86_64 219-67.el7_7.3 ubi-7 5.1 M
systemd-libs x86_64 219-67.el7_7.3 ubi-7 411 k
Transaction Summary
================================================================================
Upgrade 12 Packages
Total download size: 14 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: /var/cache/yum/x86_64/7Server/ubi-7/packages/curl-7.29.0-54.el7_7.2.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Public key for curl-7.29.0-54.el7_7.2.x86_64.rpm is not installed
--------------------------------------------------------------------------------
Total 967 kB/s | 14 MB 00:15
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Importing GPG key 0xFD431D51:
Userid : "Red Hat, Inc. (release key 2) <security@redhat.com>"
Fingerprint: 567e 347a d004 4ade 55ba 8a5f 199e 2f91 fd43 1d51
Package : redhat-release-server-7.7-10.el7.x86_64 (@anaconda/7.7)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Importing GPG key 0x2FA658E0:
Userid : "Red Hat, Inc. (auxiliary key) <security@redhat.com>"
Fingerprint: 43a6 e49c 4a38 f4be 9abf 2a53 4568 9c88 2fa6 58e0
Package : redhat-release-server-7.7-10.el7.x86_64 (@anaconda/7.7)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : nss-util-3.44.0-4.el7_7.x86_64 1/24
Updating : nss-softokn-freebl-3.44.0-8.el7_7.x86_64 2/24
Updating : systemd-libs-219-67.el7_7.3.x86_64 3/24
Updating : sqlite-3.7.17-8.el7_7.1.x86_64 4/24
Updating : nss-softokn-3.44.0-8.el7_7.x86_64 5/24
Updating : nss-sysinit-3.44.0-7.el7_7.x86_64 6/24
Updating : nss-3.44.0-7.el7_7.x86_64 7/24
Updating : libcurl-7.29.0-54.el7_7.2.x86_64 8/24
Updating : curl-7.29.0-54.el7_7.2.x86_64 9/24
Updating : systemd-219-67.el7_7.3.x86_64 10/24
Failed to get D-Bus connection: Operation not permitted
Updating : nss-tools-3.44.0-7.el7_7.x86_64 11/24
Updating : binutils-2.27-41.base.el7_7.2.x86_64 12/24
install-info: No such file or directory for /usr/share/info/as.info.gz
install-info: No such file or directory for /usr/share/info/binutils.info.gz
install-info: No such file or directory for /usr/share/info/gprof.info.gz
install-info: No such file or directory for /usr/share/info/ld.info.gz
install-info: No such file or directory for /usr/share/info/standards.info.gz
Cleanup : nss-tools-3.44.0-4.el7.x86_64 13/24
Cleanup : curl-7.29.0-54.el7_7.1.x86_64 14/24
Cleanup : systemd-219-67.el7_7.2.x86_64 15/24
Cleanup : libcurl-7.29.0-54.el7_7.1.x86_64 16/24
Cleanup : nss-sysinit-3.44.0-4.el7.x86_64 17/24
Cleanup : nss-3.44.0-4.el7.x86_64 18/24
Cleanup : nss-softokn-3.44.0-5.el7.x86_64 19/24
Cleanup : nss-softokn-freebl-3.44.0-5.el7.x86_64 20/24
Cleanup : nss-util-3.44.0-3.el7.x86_64 21/24
Cleanup : sqlite-3.7.17-8.el7.x86_64 22/24
Cleanup : systemd-libs-219-67.el7_7.2.x86_64 23/24
Cleanup : binutils-2.27-41.base.el7_7.1.x86_64 24/24
Verifying : binutils-2.27-41.base.el7_7.2.x86_64 1/24
Verifying : nss-tools-3.44.0-7.el7_7.x86_64 2/24
Verifying : nss-3.44.0-7.el7_7.x86_64 3/24
Verifying : sqlite-3.7.17-8.el7_7.1.x86_64 4/24
Verifying : systemd-libs-219-67.el7_7.3.x86_64 5/24
Verifying : nss-sysinit-3.44.0-7.el7_7.x86_64 6/24
Verifying : curl-7.29.0-54.el7_7.2.x86_64 7/24
Verifying : systemd-219-67.el7_7.3.x86_64 8/24
Verifying : libcurl-7.29.0-54.el7_7.2.x86_64 9/24
Verifying : nss-util-3.44.0-4.el7_7.x86_64 10/24
Verifying : nss-softokn-freebl-3.44.0-8.el7_7.x86_64 11/24
Verifying : nss-softokn-3.44.0-8.el7_7.x86_64 12/24
Verifying : nss-tools-3.44.0-4.el7.x86_64 13/24
Verifying : systemd-libs-219-67.el7_7.2.x86_64 14/24
Verifying : systemd-219-67.el7_7.2.x86_64 15/24
Verifying : nss-util-3.44.0-3.el7.x86_64 16/24
Verifying : curl-7.29.0-54.el7_7.1.x86_64 17/24
Verifying : libcurl-7.29.0-54.el7_7.1.x86_64 18/24
Verifying : nss-sysinit-3.44.0-4.el7.x86_64 19/24
Verifying : binutils-2.27-41.base.el7_7.1.x86_64 20/24
Verifying : nss-softokn-freebl-3.44.0-5.el7.x86_64 21/24
Verifying : nss-3.44.0-4.el7.x86_64 22/24
Verifying : nss-softokn-3.44.0-5.el7.x86_64 23/24
Verifying : sqlite-3.7.17-8.el7.x86_64 24/24
Updated:
binutils.x86_64 0:2.27-41.base.el7_7.2
curl.x86_64 0:7.29.0-54.el7_7.2
libcurl.x86_64 0:7.29.0-54.el7_7.2
nss.x86_64 0:3.44.0-7.el7_7
nss-softokn.x86_64 0:3.44.0-8.el7_7
nss-softokn-freebl.x86_64 0:3.44.0-8.el7_7
nss-sysinit.x86_64 0:3.44.0-7.el7_7
nss-tools.x86_64 0:3.44.0-7.el7_7
nss-util.x86_64 0:3.44.0-4.el7_7
sqlite.x86_64 0:3.7.17-8.el7_7.1
systemd.x86_64 0:219-67.el7_7.3
systemd-libs.x86_64 0:219-67.el7_7.3
Complete!
Loaded plugins: ovl, product-id, search-disabled-repos
Resolving Dependencies
--> Running transaction check
---> Package gcc.x86_64 0:4.8.5-39.el7 will be installed
--> Processing Dependency: cpp = 4.8.5-39.el7 for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libgomp = 4.8.5-39.el7 for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libgomp.so.1()(64bit) for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-4.8.5-39.el7.x86_64
--> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-4.8.5-39.el7.x86_64
---> Package gcc-c++.x86_64 0:4.8.5-39.el7 will be installed
--> Processing Dependency: libstdc++-devel = 4.8.5-39.el7 for package: gcc-c++-4.8.5-39.el7.x86_64
---> Package make.x86_64 1:3.82-24.el7 will be installed
--> Running transaction check
---> Package cpp.x86_64 0:4.8.5-39.el7 will be installed
---> Package glibc-devel.x86_64 0:2.17-292.el7 will be installed
--> Processing Dependency: glibc-headers = 2.17-292.el7 for package: glibc-devel-2.17-292.el7.x86_64
--> Processing Dependency: glibc-headers for package: glibc-devel-2.17-292.el7.x86_64
---> Package libgomp.x86_64 0:4.8.5-39.el7 will be installed
---> Package libmpc.x86_64 0:1.0.1-3.el7 will be installed
---> Package libstdc++-devel.x86_64 0:4.8.5-39.el7 will be installed
---> Package mpfr.x86_64 0:3.1.1-4.el7 will be installed
--> Running transaction check
---> Package glibc-headers.x86_64 0:2.17-292.el7 will be installed
--> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.17-292.el7.x86_64
--> Processing Dependency: kernel-headers for package: glibc-headers-2.17-292.el7.x86_64
--> Running transaction check
---> Package kernel-headers.x86_64 0:3.10.0-1062.12.1.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
gcc x86_64 4.8.5-39.el7 ubi-7 16 M
gcc-c++ x86_64 4.8.5-39.el7 ubi-7 7.2 M
make x86_64 1:3.82-24.el7 ubi-7 421 k
Installing for dependencies:
cpp x86_64 4.8.5-39.el7 ubi-7 6.0 M
glibc-devel x86_64 2.17-292.el7 ubi-7 1.1 M
glibc-headers x86_64 2.17-292.el7 ubi-7 687 k
kernel-headers x86_64 3.10.0-1062.12.1.el7 ubi-7 8.7 M
libgomp x86_64 4.8.5-39.el7 ubi-7 158 k
libmpc x86_64 1.0.1-3.el7 ubi-7 51 k
libstdc++-devel x86_64 4.8.5-39.el7 ubi-7 1.5 M
mpfr x86_64 3.1.1-4.el7 ubi-7 203 k
Transaction Summary
================================================================================
Install 3 Packages (+8 Dependent packages)
Total download size: 42 M
Installed size: 86 M
Downloading packages:
--------------------------------------------------------------------------------
Total 6.8 MB/s | 42 MB 00:06
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : mpfr-3.1.1-4.el7.x86_64 1/11
Installing : libmpc-1.0.1-3.el7.x86_64 2/11
Installing : cpp-4.8.5-39.el7.x86_64 3/11
Installing : kernel-headers-3.10.0-1062.12.1.el7.x86_64 4/11
Installing : glibc-headers-2.17-292.el7.x86_64 5/11
Installing : glibc-devel-2.17-292.el7.x86_64 6/11
Installing : libstdc++-devel-4.8.5-39.el7.x86_64 7/11
Installing : libgomp-4.8.5-39.el7.x86_64 8/11
Installing : gcc-4.8.5-39.el7.x86_64 9/11
Installing : gcc-c++-4.8.5-39.el7.x86_64 10/11
Installing : 1:make-3.82-24.el7.x86_64 11/11
Verifying : glibc-devel-2.17-292.el7.x86_64 1/11
Verifying : gcc-c++-4.8.5-39.el7.x86_64 2/11
Verifying : 1:make-3.82-24.el7.x86_64 3/11
Verifying : mpfr-3.1.1-4.el7.x86_64 4/11
Verifying : libgomp-4.8.5-39.el7.x86_64 5/11
Verifying : libmpc-1.0.1-3.el7.x86_64 6/11
Verifying : cpp-4.8.5-39.el7.x86_64 7/11
Verifying : gcc-4.8.5-39.el7.x86_64 8/11
Verifying : libstdc++-devel-4.8.5-39.el7.x86_64 9/11
Verifying : kernel-headers-3.10.0-1062.12.1.el7.x86_64 10/11
Verifying : glibc-headers-2.17-292.el7.x86_64 11/11
Installed:
gcc.x86_64 0:4.8.5-39.el7 gcc-c++.x86_64 0:4.8.5-39.el7
make.x86_64 1:3.82-24.el7
Dependency Installed:
cpp.x86_64 0:4.8.5-39.el7
glibc-devel.x86_64 0:2.17-292.el7
glibc-headers.x86_64 0:2.17-292.el7
kernel-headers.x86_64 0:3.10.0-1062.12.1.el7
libgomp.x86_64 0:4.8.5-39.el7
libmpc.x86_64 0:1.0.1-3.el7
libstdc++-devel.x86_64 0:4.8.5-39.el7
mpfr.x86_64 0:3.1.1-4.el7
Complete!
Loaded plugins: ovl, product-id, search-disabled-repos
Cleaning repos: ubi-7 ubi-7-rhah ubi-7-server-devtools-rpms
: ubi-7-server-extras-rpms ubi-7-server-optional-rpms
: ubi-server-rhscl-7-rpms
INFO[0080] Taking snapshot of full filesystem...
INFO[0081] Resolving paths-- build 2
$ docker run --rm --entrypoint "" -v /host/path/to/dockerfile/directory/kaniko/20200207-001//build2:/workspace gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile.debug-a1af057f997316bfb1c4d2d82719d78481a02a79 --no-push
INFO[0000] Resolved base name index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79 to index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
INFO[0000] Resolved base name index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79 to index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
INFO[0000] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
INFO[0001] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
INFO[0003] Built cross stage deps: map[]
INFO[0003] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
INFO[0003] Retrieving image manifest index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
INFO[0004] Unpacking rootfs as cmd RUN echo test requires it.
INFO[0023] Taking snapshot of full filesystem...
INFO[0023] Resolving paths
INFO[0024] RUN echo test
INFO[0024] cmd: /bin/sh
INFO[0024] args: [-c echo test]
test
INFO[0024] Taking snapshot of full filesystem...
INFO[0024] Resolving paths
INFO[0025] No files were changed, appending empty layer to config. No layer added to image.
INFO[0025] Skipping push to container registry due to --no-push flag-- Dockerfile.debug-a1af057f997316bfb1c4d2d82719d78481a02a79 used for build 2
FROM index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-a1af057f997316bfb1c4d2d82719d78481a02a79
RUN echo test
@tanguydelignieresaccenture that info is great. Thanks for confirming the fix!
diegosucaria
commented
4 years ago I see this is merged, but I have many Google Cloud Build pipelines using kaniko:latest which are still failing with the same problem..... any ideas why? should I use another tag?
Thanks
dinvlad
commented
4 years ago That's because the :latest/:debug tags were not updated yet, to the build with the fix.
yuchunyu97
commented
4 years ago error building image: error building stage: failed to get filesystem from image: removing whiteout etc/ca-certificates/.wh..wh..opq: fstatat /etc/ca-certificates/.wh..opq: operation not permittedThis issue also occurred in version v0.18.0, I don't know what happened?
pbecotte
commented
4 years ago Just to chime in, using Kaniko for the first time and am hitting a similar issue with invalid images on v0.19.0
tejal29
commented
4 years ago @pbecotte We added a fix in #1214 in v0.20.0 onwards. Can you please see if you still see the issue?
Actual behavior A clear and concise description of what the bug is.
An image, based on redhat Universal Based Image 7, installing gcc, gcc-c++ and make with yum package manager, built with kaniko v0.16.0 is usable as a FROM in another kaniko build and allows to successfully build a new sub image.
An image, based on redhat Universal Based Image 7, installing gcc, gcc-c++ and make with yum package manager, built with kaniko v0.17.1 is usable as a FROM in another kaniko build but does not allows to successfully build a new sub image and throws the following error:
error building image: error building stage: failed to get filesystem from image: removing whiteout .wh.dev: unlinkat //dev/pts/ptmx: operation not permittedExpected behavior A clear and concise description of what you expected to happen. An image, based on redhat Universal Based Image 7, installing gcc, gcc-c++ and make with yum package manager, built with kaniko v0.17.1 is usable as a FROM in another kaniko build and allows to successfully build a new sub image.
To Reproduce Steps to reproduce the behavior:
docker run --rm --entrypoint "" -v /host/path/to/kaniko/config.json:/kaniko/.docker/config.json -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build1:/workspace gcr.io/kaniko-project/executor:debug-v0.16.0 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile --destination index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.16.0output:Dependencies Resolved
================================================================================ Package Arch Version Repository Size
Updating: binutils x86_64 2.27-41.base.el7_7.2 ubi-7 5.9 M curl x86_64 7.29.0-54.el7_7.2 ubi-7 270 k libcurl x86_64 7.29.0-54.el7_7.2 ubi-7 223 k nss x86_64 3.44.0-7.el7_7 ubi-7 854 k nss-softokn x86_64 3.44.0-8.el7_7 ubi-7 330 k nss-softokn-freebl x86_64 3.44.0-8.el7_7 ubi-7 224 k nss-sysinit x86_64 3.44.0-7.el7_7 ubi-7 65 k nss-tools x86_64 3.44.0-7.el7_7 ubi-7 528 k nss-util x86_64 3.44.0-4.el7_7 ubi-7 79 k sqlite x86_64 3.7.17-8.el7_7.1 ubi-7 394 k systemd x86_64 219-67.el7_7.3 ubi-7 5.1 M systemd-libs x86_64 219-67.el7_7.3 ubi-7 411 k
Transaction Summary
Upgrade 12 Packages
Total download size: 14 M Downloading packages: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. warning: /var/cache/yum/x86_64/7Server/ubi-7/packages/curl-7.29.0-54.el7_7.2.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY Public key for curl-7.29.0-54.el7_7.2.x86_64.rpm is not installed
Total 4.0 MB/s | 14 MB 00:03 Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release Importing GPG key 0xFD431D51: Userid : "Red Hat, Inc. (release key 2) security@redhat.com" Fingerprint: 567e 347a d004 4ade 55ba 8a5f 199e 2f91 fd43 1d51 Package : redhat-release-server-7.7-10.el7.x86_64 (@anaconda/7.7) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release Importing GPG key 0x2FA658E0: Userid : "Red Hat, Inc. (auxiliary key) security@redhat.com" Fingerprint: 43a6 e49c 4a38 f4be 9abf 2a53 4568 9c88 2fa6 58e0 Package : redhat-release-server-7.7-10.el7.x86_64 (@anaconda/7.7) From : /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : nss-util-3.44.0-4.el7_7.x86_64 1/24 Updating : nss-softokn-freebl-3.44.0-8.el7_7.x86_64 2/24 Updating : systemd-libs-219-67.el7_7.3.x86_64 3/24 Updating : sqlite-3.7.17-8.el7_7.1.x86_64 4/24 Updating : nss-softokn-3.44.0-8.el7_7.x86_64 5/24 Updating : nss-sysinit-3.44.0-7.el7_7.x86_64 6/24 Updating : nss-3.44.0-7.el7_7.x86_64 7/24 Updating : libcurl-7.29.0-54.el7_7.2.x86_64 8/24 Updating : curl-7.29.0-54.el7_7.2.x86_64 9/24 Updating : systemd-219-67.el7_7.3.x86_64 10/24 Failed to get D-Bus connection: Operation not permitted Updating : nss-tools-3.44.0-7.el7_7.x86_64 11/24 Updating : binutils-2.27-41.base.el7_7.2.x86_64 12/24 install-info: No such file or directory for /usr/share/info/as.info.gz install-info: No such file or directory for /usr/share/info/binutils.info.gz install-info: No such file or directory for /usr/share/info/gprof.info.gz install-info: No such file or directory for /usr/share/info/ld.info.gz install-info: No such file or directory for /usr/share/info/standards.info.gz Cleanup : nss-tools-3.44.0-4.el7.x86_64 13/24 Cleanup : curl-7.29.0-54.el7_7.1.x86_64 14/24 Cleanup : systemd-219-67.el7_7.2.x86_64 15/24 Cleanup : libcurl-7.29.0-54.el7_7.1.x86_64 16/24 Cleanup : nss-sysinit-3.44.0-4.el7.x86_64 17/24 Cleanup : nss-3.44.0-4.el7.x86_64 18/24 Cleanup : nss-softokn-3.44.0-5.el7.x86_64 19/24 Cleanup : nss-softokn-freebl-3.44.0-5.el7.x86_64 20/24 Cleanup : nss-util-3.44.0-3.el7.x86_64 21/24 Cleanup : sqlite-3.7.17-8.el7.x86_64 22/24 Cleanup : systemd-libs-219-67.el7_7.2.x86_64 23/24 Cleanup : binutils-2.27-41.base.el7_7.1.x86_64 24/24 Verifying : binutils-2.27-41.base.el7_7.2.x86_64 1/24 Verifying : nss-tools-3.44.0-7.el7_7.x86_64 2/24 Verifying : nss-3.44.0-7.el7_7.x86_64 3/24 Verifying : sqlite-3.7.17-8.el7_7.1.x86_64 4/24 Verifying : systemd-libs-219-67.el7_7.3.x86_64 5/24 Verifying : nss-sysinit-3.44.0-7.el7_7.x86_64 6/24 Verifying : curl-7.29.0-54.el7_7.2.x86_64 7/24 Verifying : systemd-219-67.el7_7.3.x86_64 8/24 Verifying : libcurl-7.29.0-54.el7_7.2.x86_64 9/24 Verifying : nss-util-3.44.0-4.el7_7.x86_64 10/24 Verifying : nss-softokn-freebl-3.44.0-8.el7_7.x86_64 11/24 Verifying : nss-softokn-3.44.0-8.el7_7.x86_64 12/24 Verifying : nss-tools-3.44.0-4.el7.x86_64 13/24 Verifying : systemd-libs-219-67.el7_7.2.x86_64 14/24 Verifying : systemd-219-67.el7_7.2.x86_64 15/24 Verifying : nss-util-3.44.0-3.el7.x86_64 16/24 Verifying : curl-7.29.0-54.el7_7.1.x86_64 17/24 Verifying : libcurl-7.29.0-54.el7_7.1.x86_64 18/24 Verifying : nss-sysinit-3.44.0-4.el7.x86_64 19/24 Verifying : binutils-2.27-41.base.el7_7.1.x86_64 20/24 Verifying : nss-softokn-freebl-3.44.0-5.el7.x86_64 21/24 Verifying : nss-3.44.0-4.el7.x86_64 22/24 Verifying : nss-softokn-3.44.0-5.el7.x86_64 23/24 Verifying : sqlite-3.7.17-8.el7.x86_64 24/24
Updated: binutils.x86_64 0:2.27-41.base.el7_7.2 curl.x86_64 0:7.29.0-54.el7_7.2 libcurl.x86_64 0:7.29.0-54.el7_7.2 nss.x86_64 0:3.44.0-7.el7_7 nss-softokn.x86_64 0:3.44.0-8.el7_7 nss-softokn-freebl.x86_64 0:3.44.0-8.el7_7 nss-sysinit.x86_64 0:3.44.0-7.el7_7 nss-tools.x86_64 0:3.44.0-7.el7_7 nss-util.x86_64 0:3.44.0-4.el7_7 sqlite.x86_64 0:3.7.17-8.el7_7.1 systemd.x86_64 0:219-67.el7_7.3 systemd-libs.x86_64 0:219-67.el7_7.3
Complete! Loaded plugins: ovl, product-id, search-disabled-repos Resolving Dependencies --> Running transaction check ---> Package gcc.x86_64 0:4.8.5-39.el7 will be installed --> Processing Dependency: cpp = 4.8.5-39.el7 for package: gcc-4.8.5-39.el7.x86_64 --> Processing Dependency: libgomp = 4.8.5-39.el7 for package: gcc-4.8.5-39.el7.x86_64 --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc-4.8.5-39.el7.x86_64 --> Processing Dependency: libgomp.so.1()(64bit) for package: gcc-4.8.5-39.el7.x86_64 --> Processing Dependency: libmpc.so.3()(64bit) for package: gcc-4.8.5-39.el7.x86_64 --> Processing Dependency: libmpfr.so.4()(64bit) for package: gcc-4.8.5-39.el7.x86_64 ---> Package gcc-c++.x86_64 0:4.8.5-39.el7 will be installed --> Processing Dependency: libstdc++-devel = 4.8.5-39.el7 for package: gcc-c++-4.8.5-39.el7.x86_64 ---> Package make.x86_64 1:3.82-24.el7 will be installed --> Running transaction check ---> Package cpp.x86_64 0:4.8.5-39.el7 will be installed ---> Package glibc-devel.x86_64 0:2.17-292.el7 will be installed --> Processing Dependency: glibc-headers = 2.17-292.el7 for package: glibc-devel-2.17-292.el7.x86_64 --> Processing Dependency: glibc-headers for package: glibc-devel-2.17-292.el7.x86_64 ---> Package libgomp.x86_64 0:4.8.5-39.el7 will be installed ---> Package libmpc.x86_64 0:1.0.1-3.el7 will be installed ---> Package libstdc++-devel.x86_64 0:4.8.5-39.el7 will be installed ---> Package mpfr.x86_64 0:3.1.1-4.el7 will be installed --> Running transaction check ---> Package glibc-headers.x86_64 0:2.17-292.el7 will be installed --> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers-2.17-292.el7.x86_64 --> Processing Dependency: kernel-headers for package: glibc-headers-2.17-292.el7.x86_64 --> Running transaction check ---> Package kernel-headers.x86_64 0:3.10.0-1062.12.1.el7 will be installed --> Finished Dependency Resolution
Dependencies Resolved
================================================================================ Package Arch Version Repository Size
Installing: gcc x86_64 4.8.5-39.el7 ubi-7 16 M gcc-c++ x86_64 4.8.5-39.el7 ubi-7 7.2 M make x86_64 1:3.82-24.el7 ubi-7 421 k Installing for dependencies: cpp x86_64 4.8.5-39.el7 ubi-7 6.0 M glibc-devel x86_64 2.17-292.el7 ubi-7 1.1 M glibc-headers x86_64 2.17-292.el7 ubi-7 687 k kernel-headers x86_64 3.10.0-1062.12.1.el7 ubi-7 8.7 M libgomp x86_64 4.8.5-39.el7 ubi-7 158 k libmpc x86_64 1.0.1-3.el7 ubi-7 51 k libstdc++-devel x86_64 4.8.5-39.el7 ubi-7 1.5 M mpfr x86_64 3.1.1-4.el7 ubi-7 203 k
Transaction Summary
Install 3 Packages (+8 Dependent packages)
Total download size: 42 M Installed size: 86 M Downloading packages:
Total 5.8 MB/s | 42 MB 00:07 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : mpfr-3.1.1-4.el7.x86_64 1/11 Installing : libmpc-1.0.1-3.el7.x86_64 2/11 Installing : cpp-4.8.5-39.el7.x86_64 3/11 Installing : kernel-headers-3.10.0-1062.12.1.el7.x86_64 4/11 Installing : glibc-headers-2.17-292.el7.x86_64 5/11 Installing : glibc-devel-2.17-292.el7.x86_64 6/11 Installing : libstdc++-devel-4.8.5-39.el7.x86_64 7/11 Installing : libgomp-4.8.5-39.el7.x86_64 8/11 Installing : gcc-4.8.5-39.el7.x86_64 9/11 Installing : gcc-c++-4.8.5-39.el7.x86_64 10/11 Installing : 1:make-3.82-24.el7.x86_64 11/11 Verifying : glibc-devel-2.17-292.el7.x86_64 1/11 Verifying : gcc-c++-4.8.5-39.el7.x86_64 2/11 Verifying : 1:make-3.82-24.el7.x86_64 3/11 Verifying : mpfr-3.1.1-4.el7.x86_64 4/11 Verifying : libgomp-4.8.5-39.el7.x86_64 5/11 Verifying : libmpc-1.0.1-3.el7.x86_64 6/11 Verifying : cpp-4.8.5-39.el7.x86_64 7/11 Verifying : gcc-4.8.5-39.el7.x86_64 8/11 Verifying : libstdc++-devel-4.8.5-39.el7.x86_64 9/11 Verifying : kernel-headers-3.10.0-1062.12.1.el7.x86_64 10/11 Verifying : glibc-headers-2.17-292.el7.x86_64 11/11
Installed: gcc.x86_64 0:4.8.5-39.el7 gcc-c++.x86_64 0:4.8.5-39.el7 make.x86_64 1:3.82-24.el7
Dependency Installed: cpp.x86_64 0:4.8.5-39.el7 glibc-devel.x86_64 0:2.17-292.el7 glibc-headers.x86_64 0:2.17-292.el7 kernel-headers.x86_64 0:3.10.0-1062.12.1.el7 libgomp.x86_64 0:4.8.5-39.el7 libmpc.x86_64 0:1.0.1-3.el7 libstdc++-devel.x86_64 0:4.8.5-39.el7 mpfr.x86_64 0:3.1.1-4.el7
Complete! Loaded plugins: ovl, product-id, search-disabled-repos Cleaning repos: ubi-7 ubi-7-rhah ubi-7-server-devtools-rpms : ubi-7-server-extras-rpms ubi-7-server-optional-rpms : ubi-server-rhscl-7-rpms INFO[0065] Taking snapshot of full filesystem... INFO[0065] Adding whiteout for /var/lib/yum/yumdb/n/939bd19e4f12fb2195f9571120092f8ae8d57dc1-nss-softokn-freebl-3.44.0-5.el7-x86_64 INFO[0065] Adding whiteout for /var/lib/yum/rpmdb-indexes/conflicts INFO[0065] Adding whiteout for /usr/lib64/libbfd-2.27-41.base.el7_7.1.so INFO[0065] Adding whiteout for /var/lib/yum/yumdb/b/4c369229881b300e3e564b9be2679d24a74368fb-binutils-2.27-41.base.el7_7.1-x86_64 INFO[0065] Adding whiteout for /var/lib/yum/yumdb/s/7488ccedb864f45573f8f76b7cf76141f97eb35f-sqlite-3.7.17-8.el7-x86_64 INFO[0065] Adding whiteout for /var/lib/yum/rpmdb-indexes/version INFO[0065] Adding whiteout for /usr/lib64/libopcodes-2.27-41.base.el7_7.1.so INFO[0065] Adding whiteout for /var/lib/yum/yumdb/l/317aba0360ab628f9efdd4f0b4b4a313039c89a2-libcurl-7.29.0-54.el7_7.1-x86_64 INFO[0065] Adding whiteout for /var/lib/yum/yumdb/n/1d886c11b8478068d9f8bdd7764b94501635306d-nss-sysinit-3.44.0-4.el7-x86_64 INFO[0065] Adding whiteout for /var/lib/yum/rpmdb-indexes/obsoletes INFO[0065] Adding whiteout for /var/lib/yum/rpmdb-indexes/file-requires INFO[0065] Adding whiteout for /var/lib/yum/yumdb/n/9c358fa289102ab9c0d28decb5de985391517ab5-nss-3.44.0-4.el7-x86_64 INFO[0065] Adding whiteout for /var/lib/yum/yumdb/s/8ea510704120f3378d00b516e3ffbfc7ee67f66d-systemd-219-67.el7_7.2-x86_64 INFO[0065] Adding whiteout for /var/lib/yum/yumdb/n/ce3e8fdc5b9723da97371f4e65ec44e8490aa84a-nss-softokn-3.44.0-5.el7-x86_64 INFO[0065] Adding whiteout for /var/lib/yum/yumdb/n/d700496ea41582ea9596d71c2c797a7258e98284-nss-util-3.44.0-3.el7-x86_64 INFO[0065] Adding whiteout for /var/lib/yum/yumdb/c/13e4cfa6c01c54f7bd005b45b2f86e023e0bf3d9-curl-7.29.0-54.el7_7.1-x86_64 INFO[0065] Adding whiteout for /var/lib/yum/yumdb/n/fd1dd9f95ea47fbb41a1b2c62b60edefa0e491c3-nss-tools-3.44.0-4.el7-x86_64 INFO[0065] Adding whiteout for /var/lib/yum/yumdb/s/3924e2a2ddea4327fa580f5430e3fc23d283c145-systemd-libs-219-67.el7_7.2-x86_64 INFO[0065] Adding whiteout for /var/lib/yum/rpmdb-indexes/pkgtups-checksums
docker run --rm --entrypoint "" -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build2:/workspace gcr.io/kaniko-project/executor:debug-v0.17.1 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile.debug-v0.16.0 --no-pushoutput:docker run --rm --entrypoint "" -v /host/path/to/dockerfile/directory/kaniko/20200207-001/build2:/workspace gcr.io/kaniko-project/executor:debug-v0.17.1 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile.debug-v0.17.1 --no-pushoutput:Additional Information
Dockerfile "build1"
Dockerfile.debug-v0.16.0 "build2"
Dockerfile.debug-v0.17.1 "build2"
Please note:
That "Build1" build with kaniko v0.16.0 has several
Adding whiteoutnot present in build with kaniko v0.17.1"Build1"
images index.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.16.0andindex.docker.io/tanguydelignieres/kaniko_bugs_20200207-001_build1:debug-v0.17.1have been pushed to my dockerhub if you need them.I can't tell if this issue is related to #763, you may have a check too.
Triage Notes for the Maintainers
--cacheflag