Actual behavior When building a multistage image and using busybox as a shell in secondary stage works when using docker but fails when using kaniko.

Preparing to unpack .../busybox_1%3a1.30.1-4ubuntu6.4_amd64.deb ...
Unpacking busybox (1:1.30.1-4ubuntu6.4) ...
Setting up busybox (1:1.30.1-4ubuntu6.4) ...
INFO[0014] Taking snapshot of full filesystem...        
INFO[0015] Saving file usr/lib for later use            
INFO[0016] Saving file lib for later use                
INFO[0016] Saving file usr/lib64 for later use          
INFO[0016] Saving file lib64 for later use              
INFO[0016] Saving file etc for later use                
INFO[0016] Saving file usr/lib/os-release for later use 
INFO[0016] Saving file usr/lib/ssl for later use        
INFO[0016] Saving file usr/lib/locale for later use     
INFO[0016] Saving file home for later use               
INFO[0016] Saving file usr/share/ca-certificates for later use 
INFO[0016] Saving file bin/busybox for later use        
INFO[0016] Deleting filesystem...                       
INFO[0016] No base image, nothing to extract            
INFO[0016] Executing 0 build triggers                   
INFO[0016] Building stage 'scratch' [idx: '1', base-idx: '-1'] 
INFO[0016] Unpacking rootfs as cmd ADD tmp.tgz / requires it. 
INFO[0016] Using files from context: [/workspace/tmp.tgz] 
INFO[0016] ADD tmp.tgz /                                
INFO[0016] Unpacking local tar archive tmp.tgz to /     
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /lib /lib/                 
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /lib64 /lib64/             
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /etc /etc/                 
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /usr/lib/os-release /usr/lib/os-release 
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /usr/lib/ssl /usr/lib/ssl/ 
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /usr/lib/locale /usr/lib/locale/ 
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /home /home/               
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /usr/share/ca-certificates /usr/share/ca-certificates 
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /bin/busybox /bin/busybox  
INFO[0016] Taking snapshot of files...                  
INFO[0016] COPY --from=build /bin/busybox /bin/sh       
INFO[0016] Taking snapshot of files...                  
INFO[0016] RUN  /bin/busybox ln -s /bin/busybox /bin/ln; ln -s /bin/busybox /bin/rm  ; for i in $(/bin/busybox --list); do if [ -f /bin/$i ]; then /bin/busybox rm /bin/$i; fi; if [ ! -L /bin/$i ]; then /bin/busybox ln -s /bin/busybox /bin/$i;fi;done;mkdir -p /usr/bin;/bin/busybox ln -s /bin/busybox /usr/bin/env 
INFO[0016] Initializing snapshotter ...                 
INFO[0016] Taking snapshot of full filesystem...        
INFO[0016] Cmd: /bin/sh                                 
INFO[0016] Args: [-c /bin/busybox ln -s /bin/busybox /bin/ln; ln -s /bin/busybox /bin/rm  ; for i in $(/bin/busybox --list); do if [ -f /bin/$i ]; then /bin/busybox rm /bin/$i; fi; if [ ! -L /bin/$i ]; then /bin/busybox ln -s /bin/busybox /bin/$i;fi;done;mkdir -p /usr/bin;/bin/busybox ln -s /bin/busybox /usr/bin/env] 
INFO[0016] Running: [/bin/sh -c /bin/busybox ln -s /bin/busybox /bin/ln; ln -s /bin/busybox /bin/rm  ; for i in $(/bin/busybox --list); do if [ -f /bin/$i ]; then /bin/busybox rm /bin/$i; fi; if [ ! -L /bin/$i ]; then /bin/busybox ln -s /bin/busybox /bin/$i;fi;done;mkdir -p /usr/bin;/bin/busybox ln -s /bin/busybox /usr/bin/env] 
error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: no such file or directory

Expected behavior Build to work like it does with Docker

To Reproduce Steps to reproduce the behavior:

Create docker file
create bash script kaniko_link_fixer.sh to avoid issue with symlinks and kaniko
Build with "docker run -it --rm -v ${PWD}:/workspace -v "$HOME/.config/gcloud:/root/.config/gcloud" gcr.io/kaniko-project/executor:latest --no-push"

Additional Information

Dockerfile


FROM gcr.io/cloud-marketplace/google/ubuntu2004 as build
COPY kaniko_link_fixer.sh .
RUN  apt-get update && \
apt-get install -y \
 --no-install-recommends busybox && \
./kaniko_link_fixer.sh

FROM scratch ADD tmp.tgz /

create a basic set up so linux will work

COPY --from=build /lib /lib/ COPY --from=build /lib64 /lib64/

some basic data files

COPY --from=build /etc /etc/ COPY --from=build /usr/lib/os-release /usr/lib/os-release COPY --from=build /usr/lib/ssl /usr/lib/ssl/ COPY --from=build /usr/lib/locale /usr/lib/locale/

COPY --from=build /usr/local/bin/grpc /usr/local/bin/grpc/

COPY --from=build /home /home/ COPY --from=build /usr/share/ca-certificates /usr/share/ca-certificates

debug can be removed in non debug

COPY --from=build /bin/busybox /bin/busybox COPY --from=build /bin/busybox /bin/sh

RUN /bin/busybox ln -s /bin/busybox /bin/ln; ln -s /bin/busybox /bin/rm ; for i in $(/bin/busybox --list); do if [ -f /bin/$i ]; then /bin/busybox rm /bin/$i; fi; if [ ! -L /bin/$i ]; then /bin/busybox ln -s /bin/busybox /bin/$i;fi;done;mkdir -p /usr/bin;/bin/busybox ln -s /bin/busybox /usr/bin/env

 - Build Context
   kaniko_link_fixer.sh used to remove symlinks and replace with files so copy down works

!/usr/bin/env bash

so kaniko breaks when you copy to second layers in multi-stage build

https://github.com/GoogleContainerTools/kaniko/issues/2381

this is hopefully a work around

basically this resolves symlinks for specific directories that block multi stage

find links that point at directories

replace the directory links with directory

copy cntents recursively into new directory

we do this stage first to get to just file links

only do this if running in kaniko detected directory kaniko exists

if [ -d /kaniko ] then for f in $(find /usr/lib/ssl /etc/ssl -type l ) do export a=$(/usr/bin/readlink -f $f) if [ -d "$f" ] then rm "$f" mkdir -p "$f" cp -r "$a" "$f" fi done

now we should just have file links we realise the links to the actual files

this does duplicate data so costs disk space

for f in $(find /usr/lib/ssl /etc/ssl -type l ) do export a=$(/usr/bin/readlink -f $f) if [ -f "$f" ] then rm "$f" cp "$a" "$f" else echo "should not happen!!!!!!!!!! $f is not a file" fi done fi


You will also need a tar tmp.tgz contents not that important in my case I have directory /tmp with 1777 permissions to provide a /tmp
 - Kaniko Image (fully qualified with digest)
 gcr.io/kaniko-project/executor:latest
 sha256:23ae6ccaba2b0f599966dbc5ecf38aa4404f4cd799add224167eaf285696551a

 **Triage Notes for the Maintainers**
 <!-- 🎉🎉🎉 Thank you for an opening an issue !!! 🎉🎉🎉
We are doing our best to get to this. Please help us by helping us prioritize your issue by filling the section below -->

 | **Description** | **Yes/No** |
 |----------------|---------------|
 | Please check if this a new feature you are proposing        | <ul><li>- [ ] </li></ul>|
 | Please check if the build works in docker but not in kaniko | <ul><li>- [ ] </li></ul>| 
 | Please check if this error is seen when you use `--cache` flag | <ul><li>- [ ] </li></ul>|
 | Please check if your dockerfile is a multistage dockerfile | <ul><li>- [ ] </li></ul>|

GoogleContainerTools / kaniko

multistage build enabling and using busybox to enable RUN fails in multi stage build works with docker #2383