Installed package bash not available in produced image (alpine/apk)

Actual behavior I'm using Kaniko v1.8.1 to build an image using a Dockerfile. However I am using a custom Image in order to perform other tasks beforehand (for example I need bash and curl inside the build container). Therefore the Dockerfile for my build container image looks like this (XXX are company mirrors of either gcr or docker-hub):

FROM gcr-docker-remote.XXX/kaniko-project/executor:debug@sha256:3bc3f3a05f803cac29164ce12617a7be64931748c944f6c419565f500b65e8db as kaniko
FROM docker-hub-remote.XXX/alpine@sha256:686d8c9dfa6f3ccfc8230bc3178d23f84eeaf7e457f36f271ab1acc53015037c

COPY --from=kaniko /kaniko /kaniko

RUN apk update && apk upgrade && apk add --no-cache curl bash

SHELL ["/bin/bash", "-c"]

ENV PATH=/kaniko:$PATH
ENV DOCKER_CONFIG='/kaniko/.docker'
ENV DOCKER_CREDENTIAL_GCR_CONFIG=/kaniko/.config/gcloud/docker_credential_gcr_config.json
ENV SSL_CERT_DIR=/kaniko/ssl/certs

This all seems to work fine and dandy. However when using this image to build other images, something weird happens.

I could reproduce the issue with a minimal Dockerfile, which looks like this:

FROM alpine:3.16

USER root

RUN apk add --no-cache bash git

USER 1000

SHELL ["/bin/bash", "-c"]

The image is built correctly and judging by the logoutput, the packages get installed correctly (I've removed company urls/mirrors):

INFO[0000] Retrieving image manifest XXX/alpine:3.16 
INFO[0000] Retrieving image XXX/alpine:3.16 from registry XXX 
INFO[0000] Built cross stage deps: map[]                
INFO[0000] Retrieving image manifest XXX/alpine:3.16 
INFO[0000] Returning cached image manifest              
INFO[0000] Executing 0 build triggers                   
INFO[0000] Unpacking rootfs as cmd RUN apk add --no-cache bash requires it. 
INFO[0000] USER root                                    
INFO[0000] cmd: USER                                    
INFO[0000] RUN apk add --no-cache bash                  
INFO[0000] Taking snapshot of full filesystem...        
INFO[0000] cmd: /bin/sh                                 
INFO[0000] args: [-c apk add --no-cache bash]           
INFO[0000] util.Lookup returned: &{Uid:0 Gid:0 Username:root Name:root HomeDir:/root} 
INFO[0000] performing slow lookup of group ids for root 
INFO[0000] Running: [/bin/sh -c apk add --no-cache bash] 
fetch XXX/alpine-generic-remote/v3.16/main/x86_64/APKINDEX.tar.gz
fetch XXX/alpine-generic-remote/v3.16/community/x86_64/APKINDEX.tar.gz
fetch XXX/alpine-generic-remote/edge/community/x86_64/APKINDEX.tar.gz
(1/4) Installing ncurses-terminfo-base (6.3_p20220521-r0)
(2/4) Installing ncurses-libs (6.3_p20220521-r0)
(3/4) Installing readline (8.1.2-r0)
(4/4) Installing bash (5.1.16-r2)
Executing bash-5.1.16-r2.post-install
Executing busybox-1.35.0-r14.trigger
OK: 8 MiB in 19 packages
INFO[0002] Taking snapshot of full filesystem...        
INFO[0002] USER 1000                               
INFO[0002] cmd: USER                                    
INFO[0002] SHELL ["/bin/bash", "-c"]                    
INFO[0002] Pushing image to XXX/debug-kaniko:13327563

However, if I try to run the image, there is no bash executable to be found anywhere on the system. The output of apk under /lib/apk/db/installed states:

C:Q1owNdUcG0hf9pE5c2MbsYhrwqQGE=
P:bash
V:5.1.16-r2
A:x86_64
S:435099
I:1339392
T:The GNU Bourne Again shell
U:https://www.gnu.org/software/bash/bash.html
L:GPL-3.0-or-later
o:bash
m:Natanael Copa <ncopa@alpinelinux.org>
t:1649795196
c:66da74397b67b6200e7fe9262a14544b0836a1c1
D:/bin/sh so:libc.musl-x86_64.so.1 so:libreadline.so.8
p:cmd:bash=5.1.16-r2
F:bin
R:bash
a:0:0:755
Z:Q1+dNR4F1r7bl/bUBM8Xq3d1LndOk=
...

So it looks like it's been installed, but the executable is simply missing. The git package however, which has been installed using the same command, is available and working.

Expected behavior Bash is available in $PATH after installation using apk

To Reproduce Steps to reproduce the behavior:

Build custom kaniko image
Copy the Dockerfile given above
Build it using custom kaniko image
Verify if bash has been installed by running the image

Additional Information

Dockerfile

FROM alpine:3.16

USER root

RUN apk add --no-cache bash git

USER 1000

SHELL ["/bin/bash", "-c"]

Build Context None

Kaniko Image (fully qualified with digest) gcr.io/kaniko-project/executor:debug@sha256:3bc3f3a05f803cac29164ce12617a7be64931748c944f6c419565f500b65e8db

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [ ]
Please check if this error is seen when you use `--cache` flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [ ]

For anyone else wondering, you can bypass the problem by doing the following in your custom builder image:

RUN mv /bin/bash /kaniko/bash && ln -sf /kaniko/bash /bin/sh  # symlink is necessary!
SHELL ["/kaniko/bash","-c"]

When building with the image resulting from that, bash seems to be properly installed and working in the target image.

TLDR; You need to put any binaries you add to your builder image inside the /kaniko dir, because otherwise if you try to install the same packages in your actual image build, kaniko will remove them after the build has completed. So just put the binaries into the kaniko dir (and maybe export your PATH to look into the kaniko dir).

For anyone else wondering, you can bypass the problem by doing the following in your custom builder image:
RUN mv /bin/bash /kaniko/bash && ln -sf /kaniko/bash /bin/sh  # symlink is necessary!
SHELL ["/kaniko/bash","-c"]
When building with the image resulting from that, bash seems to be properly installed and working in the target image.

TLDR; You need to put any binaries you add to your builder image inside the /kaniko dir, because otherwise if you try to install the same packages in your actual image build, kaniko will remove them after the build has completed. So just put the binaries into the kaniko dir (and maybe export your PATH to look into the kaniko dir).

Thanks for the workaround! The binaries in /kaniko were getting ignored by my builder image so I had to put them in /root/bin and add that dir to PATH. This is with kaniko v1.9.0:

builder image:

FROM alpine:3.14

...

RUN apk update && apk upgrade --no-cache && apk add --no-cache openssh-client bash
RUN mkdir /root/bin

ENV PATH="$PATH:/root/bin"

RUN mv /usr/bin/bash /root/bin/bash && ln -sf /root/bin/bash /bin/sh; \
    mv /usr/bin/ssh /root/bin/ssh;

ENTRYPOINT ["/root/bin/bash"]

target image:

FROM node:16.13-alpine3.14

RUN apk update && apk upgrade --no-cache && apk add --no-cache openssh-client git xvfb-run

...

We've had to employ this workaround, this is a super strange issue no? Like Kaniko is changing the alpine base image itself? Does anyone know more about the cause of this?

When I try to repro this with the latest version of Kaniko - v1.12.1 I am seeing bash installed in the final image:

Dockerfile

FROM alpine:3.16

USER root

RUN apk add --no-cache bash git

USER 1000

SHELL ["/bin/bash", "-c"]

aprindle@aprindle-ssd ~/kaniko  [aprindle-debug]./run_in_docker.sh /workspace/Dockerfile /usr/local/google/home/aprindle/kaniko-ctx-dir gcr.io/aprindle-test-cluster/kaniko-test:latest false |& tee ~/out.txt
INFO[0000] Retrieving image manifest alpine:3.16        
INFO[0000] Retrieving image alpine:3.16 from registry index.docker.io 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest alpine:3.16        
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Building stage 'alpine:3.16' [idx: '0', base-idx: '-1'] 
INFO[0001] Unpacking rootfs as cmd RUN apk add --no-cache bash git requires it. 
INFO[0001] USER root                                    
INFO[0001] Cmd: USER                                    
INFO[0001] RUN apk add --no-cache bash git              
INFO[0001] Initializing snapshotter ...                 
INFO[0001] Taking snapshot of full filesystem...        
INFO[0001] Cmd: /bin/sh                                 
INFO[0001] Args: [-c apk add --no-cache bash git]       
INFO[0001] Util.Lookup returned: &{Uid:0 Gid:0 Username:root Name: HomeDir:/root} 
INFO[0001] Performing slow lookup of group ids for root 
INFO[0001] Running: [/bin/sh -c apk add --no-cache bash git] 
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
(1/11) Installing ncurses-terminfo-base (6.3_p20220521-r1)
(2/11) Installing ncurses-libs (6.3_p20220521-r1)
(3/11) Installing readline (8.1.2-r0)
(4/11) Installing bash (5.1.16-r2)
Executing bash-5.1.16-r2.post-install
(5/11) Installing ca-certificates (20230506-r0)
(6/11) Installing brotli-libs (1.0.9-r6)
(7/11) Installing nghttp2-libs (1.47.0-r0)
(8/11) Installing libcurl (8.1.2-r0)
(9/11) Installing expat (2.5.0-r0)
(10/11) Installing pcre2 (10.40-r0)
(11/11) Installing git (2.36.6-r0)
Executing busybox-1.35.0-r17.trigger
Executing ca-certificates-20230506-r0.trigger
OK: 22 MiB in 25 packages
INFO[0002] Taking snapshot of full filesystem...        
INFO[0002] USER 1000                                    
INFO[0002] Cmd: USER                                    
INFO[0002] SHELL ["/bin/bash", "-c"]                    
INFO[0002] Pushing image to gcr.io/aprindle-test-cluster/kaniko-test:latest 
INFO[0005] Pushed gcr.io/aprindle-test-cluster/kaniko-test@sha256:7d04a6c6496d6a929863636499029493fe005631a56164483715fffc389bfe43

aprindle@aprindle-ssd ~/kaniko  [aprindle-debug]docker run -it gcr.io/aprindle-test-cluster/kaniko-test:latest bash
bash-5.1$ which bash
/bin/bash
bash-5.1$ env
CHARSET=UTF-8
HOSTNAME=1d2a20a04259
PWD=/
HOME=/
LANG=C.UTF-8
TERM=xterm
SHLVL=1
LC_COLLATE=C
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env
bash-5.1$

Can someone describe the exact steps they are using to reproduce missing files or verify if this is still an issue in the latest version of kaniko? Thanks

@aaron-prindle Are you sure you've used kaniko inside a custom builder image? As this is the underlying cause, kaniko itself in the officially provided images does not have this issue. However some use-cases need images with more dependencies such as bash as a shell or curl for retrieving build-time dependencies.

custom kaniko builder image can not add any packages. as the same package will not been identified as changed in Taking snapshot of full filesystem

to be more specific. if you add bash in custom kaniko builder image, on target image building phase, bash won't be identified changed, therefor bash will not be add to target image filesystem.

GoogleContainerTools / kaniko

Installed package bash not available in produced image (alpine/apk) #2177