Open ghost opened 2 years ago
Except that I am then getting the following error when trying to use this image in a subsequent build step:
INFO[0001] Unpacking rootfs as cmd RUN ... requires it.
error building image: error building stage: failed to get filesystem from image: removing whiteout .wh.proc: unlinkat //proc/fb: permission denied
Hi isn't #1724 not solving your problem?
Copying /sys
should be solved in https://github.com/GoogleContainerTools/kaniko/pull/1724/commits/d43b732b6326418e1df3c2f88165bb351fe98d99
mkdir -p /tmp/123
cat > /tmp/123/Dockerfile <<EOT
FROM debian:buster-slim AS importer
RUN apt-get update && apt-get install -y libguestfs-tools wget xz-utils
RUN wget -nv https://raspi.debian.net/verified/20210718_raspi_4_buster.img.xz && xz -d 20210718_raspi_4_buster.img.xz && mkdir /rootfs && guestfish --ro -a 20210718_raspi_4_buster.img -m /dev/sda2 copy-out / /rootfs/ && rm 20210718_raspi_4_buster.img
FROM scratch
COPY --from=importer /rootfs/ /
EOT
docker run -ti --rm -v /tmp/123:/context ghcr.io/kvaps/kaniko-executor:v1.6.0 -c /context --no-push
I use the same approach to build VM images out of docker images :) https://github.com/kvaps/opennebula-images/tree/da5bcbadbe2915167ef74fde7166ff4d75fb1151/new
So running the following (note: the download URL slightly changed)
mkdir -p /tmp/123
cat > /tmp/123/Dockerfile <<EOT
FROM debian:buster-slim AS importer
RUN apt-get update && apt-get install -y libguestfs-tools wget xz-utils
RUN wget -nv https://raspi.debian.net/tested/20210718_raspi_4_buster.img.xz && xz -d 20210718_raspi_4_buster.img.xz && mkdir /rootfs && guestfish --ro -a 20210718_raspi_4_buster.img -m /dev/sda2 copy-out / /rootfs/ && rm 20210718_raspi_4_buster.img
FROM scratch
COPY --from=importer /rootfs/ /
EOT
docker run -ti --rm -v /tmp/123:/context ghcr.io/kvaps/kaniko-executor:v1.7.0 -c /context --no-push
hangs for me at Saving file rootfs for later use
. Same for kaniko-executor:v1.6.0. If I remeber correctly running it with kaniko-executor:v1.6.0-fix back then actually worked around the hang (see above). Did that work for you?
I struggeled using your image in our Gitlab CI/CD pipeline. I digged a little deeper and it turns out that multiple achitectures are mixed up there. When pulling from x64 I obtain an arm64 image where for example /busybox/sh is indeed arm64 but the executor is x64:
uname -m
# x86_64
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
mkdir -p /tmp/123
docker run --rm -it -v /tmp/123:/context --entrypoint /busybox/sh ghcr.io/kvaps/kaniko-executor:v1.7.0-debug
# WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
# cp /busybox/sh /context/
# cp /kaniko/executor /context/
# exit
file /tmp/123/sh
# sh: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, stripped
file /tmp/123/executor
# executor: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=7BkQGRwXsNnIqlKjVroQ/0m_j6BzaX-FhkPpGcJaR/-9sTmtlaXltjhcGsg2mS/HMbpOMzbZstMMB7jfvs1, stripped
Oh that's bad, I just bought a new Macbook with M1 chip, and get confused by amount of architectures π
Images are fixed now:
ghcr.io/kvaps/kaniko-executor:v1.7.0@sha256:63c8fddec65e5b48ab41657da537f9f2360d6fbc179ecc096c414597d739143a
ghcr.io/kvaps/kaniko-executor:v1.7.0-debug@sha256:2124a5418d126d8684ed03e539b08d1b4bc5f1cc23751abeff182802cb7906ad
ghcr.io/kvaps/kaniko-warmer:v1.7.0@sha256:a3f95ed186b37cf792f5dd0f648d8b2ba4907ddb8dffee3ab0cee97a01dbeb45
The same images are used in my kubectl-build plugin
Oh that's bad, I just bought a new Macbook with M1 chip, and get confused by amount of architectures π
Images are fixed now:
ghcr.io/kvaps/kaniko-executor:v1.7.0@sha256:63c8fddec65e5b48ab41657da537f9f2360d6fbc179ecc096c414597d739143a ghcr.io/kvaps/kaniko-executor:v1.7.0-debug@sha256:2124a5418d126d8684ed03e539b08d1b4bc5f1cc23751abeff182802cb7906ad ghcr.io/kvaps/kaniko-warmer:v1.7.0@sha256:a3f95ed186b37cf792f5dd0f648d8b2ba4907ddb8dffee3ab0cee97a01dbeb45
The same images are used in my kubectl-build plugin
Unfortunately it doesn't work either.
I tried this example:
FROM registry.access.redhat.com/ubi9 AS ubi-micro-build
RUN mkdir -p /mnt/rootfs
RUN dnf install --installroot /mnt/rootfs curl --releasever 9 --setopt install_weak_deps=false --nodocs -y; dnf --installroot /mnt/rootfs clean all
FROM quay.io/keycloak/keycloak
COPY --from=ubi-micro-build /mnt/rootfs /
Oh that's bad, I just bought a new Macbook with M1 chip, and get confused by amount of architectures π Images are fixed now:
ghcr.io/kvaps/kaniko-executor:v1.7.0@sha256:63c8fddec65e5b48ab41657da537f9f2360d6fbc179ecc096c414597d739143a ghcr.io/kvaps/kaniko-executor:v1.7.0-debug@sha256:2124a5418d126d8684ed03e539b08d1b4bc5f1cc23751abeff182802cb7906ad ghcr.io/kvaps/kaniko-warmer:v1.7.0@sha256:a3f95ed186b37cf792f5dd0f648d8b2ba4907ddb8dffee3ab0cee97a01dbeb45
The same images are used in my kubectl-build plugin
Unfortunately it doesn't work either.
I tried this example:
FROM registry.access.redhat.com/ubi9 AS ubi-micro-build RUN mkdir -p /mnt/rootfs RUN dnf install --installroot /mnt/rootfs curl --releasever 9 --setopt install_weak_deps=false --nodocs -y; dnf --installroot /mnt/rootfs clean all FROM quay.io/keycloak/keycloak COPY --from=ubi-micro-build /mnt/rootfs /
Heya, had the same issue with our Keycloak container
Worked around it with the following COPY
statements instead of the full /
COPY
COPY --from=ubi-micro-build /mnt/rootfs/usr/bin/curl /usr/bin/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/* /usr/lib64/
Hope that helps :blush:
Oh that's bad, I just bought a new Macbook with M1 chip, and get confused by amount of architectures π Images are fixed now:
ghcr.io/kvaps/kaniko-executor:v1.7.0@sha256:63c8fddec65e5b48ab41657da537f9f2360d6fbc179ecc096c414597d739143a ghcr.io/kvaps/kaniko-executor:v1.7.0-debug@sha256:2124a5418d126d8684ed03e539b08d1b4bc5f1cc23751abeff182802cb7906ad ghcr.io/kvaps/kaniko-warmer:v1.7.0@sha256:a3f95ed186b37cf792f5dd0f648d8b2ba4907ddb8dffee3ab0cee97a01dbeb45
The same images are used in my kubectl-build plugin
Unfortunately it doesn't work either. I tried this example:
FROM registry.access.redhat.com/ubi9 AS ubi-micro-build RUN mkdir -p /mnt/rootfs RUN dnf install --installroot /mnt/rootfs curl --releasever 9 --setopt install_weak_deps=false --nodocs -y; dnf --installroot /mnt/rootfs clean all FROM quay.io/keycloak/keycloak COPY --from=ubi-micro-build /mnt/rootfs /
Heya, had the same issue with our Keycloak container
Worked around it with the following
COPY
statements instead of the full/
COPYCOPY --from=ubi-micro-build /mnt/rootfs/usr/bin/curl /usr/bin/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/ /lib64/
Hope that helps π
Hi, I had same issue while trying to copy curl while building keycloak image. After I did work around like your suggestion then it passed for build image however there is error while deploy pod and below is error describe pod. Did you have same issue and any idea to resolve this? Thanks!
Warning Unhealthy 32s (x10 over 50s) kubelet Liveness probe failed: curl: error while loading shared libraries: libcurl.so.4: cannot open shared object file: No such file or directory
Warning Unhealthy 30s (x11 over 50s) kubelet Readiness probe failed: curl: error while loading shared libraries: libcurl.so.4: cannot open shared object file: No such file or directory
Heya, I made an oopsie when copying my suggestion, my bad! :see_no_evil:
Give the following a try:
COPY --from=ubi-micro-build /mnt/rootfs/usr/bin/curl /usr/bin/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/* /usr/lib64/
Heya, I made an oopsie when copying my suggestion, my bad! π
Give the following a try:
COPY --from=ubi-micro-build /mnt/rootfs/usr/bin/curl /usr/bin/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/* /usr/lib64/
Tested like this
This gives me the following error:
INFO[0035] COPY --from=ubi-micro-build /mnt/rootfs/lib64/* /usr/lib64/
INFO[0035] Resolving srcs [/mnt/rootfs/lib64/*]...
error building image: error building stage: failed to execute command: resolving dest symlink: failed to eval symlinks: EvalSymlinks: too many links
I am installing curl using the following command
RUN dnf install --installroot /mnt/rootfs curl --releasever 9 --setopt install_weak_deps=false --nodocs -y && \
dnf --installroot /mnt/rootfs clean all && \
rpm --root /mnt/rootfs -e --nodeps setup
Here's my Dockerfile (with some unrelated things stripped out). Hope this helps
FROM registry.access.redhat.com/ubi9 AS ubi-micro-build
RUN mkdir -p /mnt/rootfs
RUN dnf install --installroot /mnt/rootfs curl --releasever 9 --setopt install_weak_deps=false --nodocs -y; dnf --installroot /mnt/rootfs clean all
FROM quay.io/keycloak/keycloak:22.0.2 as builder
WORKDIR /opt/keycloak
COPY --from=providergenerator /providers/bin/ /opt/keycloak/providers/
RUN /opt/keycloak/bin/kc.sh build
FROM quay.io/keycloak/keycloak:22.0.2
COPY --from=ubi-micro-build /mnt/rootfs/usr/bin/curl /usr/bin/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/* /usr/lib64/
COPY --from=builder /opt/keycloak /opt/keycloak
ENTRYPOINT ["/opt/keycloak/bin/kc.sh", "start"]
Here's my Dockerfile (with some unrelated things stripped out). Hope this helps
FROM registry.access.redhat.com/ubi9 AS ubi-micro-build RUN mkdir -p /mnt/rootfs RUN dnf install --installroot /mnt/rootfs curl --releasever 9 --setopt install_weak_deps=false --nodocs -y; dnf --installroot /mnt/rootfs clean all FROM quay.io/keycloak/keycloak:22.0.2 as builder WORKDIR /opt/keycloak COPY --from=providergenerator /providers/bin/ /opt/keycloak/providers/ RUN /opt/keycloak/bin/kc.sh build FROM quay.io/keycloak/keycloak:22.0.2 COPY --from=ubi-micro-build /mnt/rootfs/usr/bin/curl /usr/bin/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/* /usr/lib64/ COPY --from=builder /opt/keycloak /opt/keycloak ENTRYPOINT ["/opt/keycloak/bin/kc.sh", "start"]
Thank you for the replying. I faced the same issue @sanderjochems-whyellow posted above.
Could you please share which version of kaniko that you are using?
Mine is gcr.io/kaniko-project/executor:v1.5.1-debug and I have tried with lastest version gcr.io/kaniko-project/executor:v1.17.0-debug that are raised error
INFO[0075] Resolving srcs [/mnt/rootfs/lib64/*]...
error building image: error building stage: failed to execute command: resolving dest symlink: failed to eval symlinks: EvalSymlinks: too many links
Could you please share which version of kaniko that you are using?
I am using version gcr.io/kaniko-project/executor:v1.9.0-debug
which is also not working.
I am building the Dockerfile inside of a AWS EKS (Kubernetes) cluster on Fargate (AWS Serverless compute) pods. They are running without privileged mode.
Maybe try like this? It's a bit verbose but should work around the symlink issue. I did notice that our Kaniko version is wildly outdated, might need to look into upgrading that :sweat_smile: gcr.io/kaniko-project/executor:debug-v0.22.0
FROM quay.io/keycloak/keycloak:22.0.2
COPY --from=ubi-micro-build /mnt/rootfs/usr/bin/curl /usr/bin/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcurl.so.4 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcurl.so.4.7.0 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libnghttp2.so.14 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libidn2.so.0 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libssh.so.4 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libpsl.so.5 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libssl.so.3 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcrypto.so.3 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libgssapi_krb5.so.2 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libkrb5.so.3 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libk5crypto.so.3 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcom_err.so.2 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libldap.so.2 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/liblber.so.2 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libbrotlidec.so.1 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libunistring.so.2 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libkrb5support.so.0 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libkeyutils.so.1 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libevent-2.1.so.7 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libsasl2.so.3 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libbrotlicommon.so.1 /usr/lib64/
COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcrypt.so.2 /usr/lib64/
Depending on versions of stuff, you might need to copy over more or less of these .so files. I found the list for my instance like this
$ docker run -it kc '--&& ldd /usr/bin/curl'
linux-vdso.so.1 (0x00007ffc9fbdd000)
libcurl.so.4 => /lib64/libcurl.so.4 (0x00007fbd74599000)
libz.so.1 => /lib64/libz.so.1 (0x00007fbd7457f000)
libc.so.6 => /lib64/libc.so.6 (0x00007fbd74376000)
libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007fbd7434c000)
libidn2.so.0 => /lib64/libidn2.so.0 (0x00007fbd7432b000)
libssh.so.4 => /lib64/libssh.so.4 (0x00007fbd742b7000)
libpsl.so.5 => /lib64/libpsl.so.5 (0x00007fbd742a1000)
libssl.so.3 => /lib64/libssl.so.3 (0x00007fbd741fb000)
libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007fbd73dca000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fbd73d73000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fbd73c98000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fbd73c7f000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fbd73c76000)
libldap.so.2 => /lib64/libldap.so.2 (0x00007fbd73c0f000)
liblber.so.2 => /lib64/liblber.so.2 (0x00007fbd73bfd000)
libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007fbd73bef000)
/lib64/ld-linux-x86-64.so.2 (0x00007fbd7467d000)
libunistring.so.2 => /lib64/libunistring.so.2 (0x00007fbd73a6a000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fbd73a57000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fbd73a50000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fbd73a3c000)
libevent-2.1.so.7 => /lib64/libevent-2.1.so.7 (0x00007fbd739e3000)
libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007fbd739c3000)
libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007fbd7399e000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fbd73971000)
libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007fbd73937000)
libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007fbd7389b000)
Maybe try like this? It's a bit verbose but should work around the symlink issue. I did notice that our Kaniko version is wildly outdated, might need to look into upgrading that π
gcr.io/kaniko-project/executor:debug-v0.22.0
FROM quay.io/keycloak/keycloak:22.0.2 COPY --from=ubi-micro-build /mnt/rootfs/usr/bin/curl /usr/bin/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcurl.so.4 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcurl.so.4.7.0 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libnghttp2.so.14 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libidn2.so.0 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libssh.so.4 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libpsl.so.5 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libssl.so.3 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcrypto.so.3 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libgssapi_krb5.so.2 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libkrb5.so.3 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libk5crypto.so.3 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcom_err.so.2 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libldap.so.2 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/liblber.so.2 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libbrotlidec.so.1 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libunistring.so.2 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libkrb5support.so.0 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libkeyutils.so.1 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libevent-2.1.so.7 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libsasl2.so.3 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libbrotlicommon.so.1 /usr/lib64/ COPY --from=ubi-micro-build /mnt/rootfs/lib64/libcrypt.so.2 /usr/lib64/
Depending on versions of stuff, you might need to copy over more or less of these .so files. I found the list for my instance like this
$ docker run -it kc '--&& ldd /usr/bin/curl' linux-vdso.so.1 (0x00007ffc9fbdd000) libcurl.so.4 => /lib64/libcurl.so.4 (0x00007fbd74599000) libz.so.1 => /lib64/libz.so.1 (0x00007fbd7457f000) libc.so.6 => /lib64/libc.so.6 (0x00007fbd74376000) libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007fbd7434c000) libidn2.so.0 => /lib64/libidn2.so.0 (0x00007fbd7432b000) libssh.so.4 => /lib64/libssh.so.4 (0x00007fbd742b7000) libpsl.so.5 => /lib64/libpsl.so.5 (0x00007fbd742a1000) libssl.so.3 => /lib64/libssl.so.3 (0x00007fbd741fb000) libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007fbd73dca000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fbd73d73000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fbd73c98000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fbd73c7f000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fbd73c76000) libldap.so.2 => /lib64/libldap.so.2 (0x00007fbd73c0f000) liblber.so.2 => /lib64/liblber.so.2 (0x00007fbd73bfd000) libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007fbd73bef000) /lib64/ld-linux-x86-64.so.2 (0x00007fbd7467d000) libunistring.so.2 => /lib64/libunistring.so.2 (0x00007fbd73a6a000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fbd73a57000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fbd73a50000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fbd73a3c000) libevent-2.1.so.7 => /lib64/libevent-2.1.so.7 (0x00007fbd739e3000) libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007fbd739c3000) libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007fbd7399e000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fbd73971000) libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007fbd73937000) libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007fbd7389b000)
Hi @niekcandaele,
Thank you a lot for support. We are using keycloak operator to generate keycloak instance. I just updated keycloak docker image and it requires to update keycloak operator to 22.0.4 as well (I forgot to update this one) and it is using http-get to do health check (https://www.keycloak.org/server/health#_kubernetes), then no need install curl anymore and no need to deal with install stuff currently. Thanks!
Dockerfile to reproduce:
For v1.6.0 this hangs at "Saving file rootfs for later use". This is also described in #960 and the PR #1724 improves the situation. However, it now stops in the copy step reporting
The workaround from #1007 seems to work though.