yarn is missing files after container is built with kaniko

[drptbl]

Dockerfile with yarn is missing files after container is built with kaniko, tried on 0.17.1 and 0.16.0:

FROM node:10.19.0-buster
ENV YARN_VERSION 1.22.0

# install latest yarn
RUN curl -fSLO --compressed "https://yarnpkg.com/downloads/$YARN_VERSION/yarn-v$YARN_VERSION.tar.gz" \
    && tar -xzf yarn-v$YARN_VERSION.tar.gz -C /opt/ \
    && ln -snf /opt/yarn-v$YARN_VERSION/bin/yarn /usr/local/bin/yarn \
    && ln -snf /opt/yarn-v$YARN_VERSION/bin/yarnpkg /usr/local/bin/yarnpkg \
    && rm yarn-v$YARN_VERSION.tar.gz

.. now if you pull this image and cd /opt/yarn-v1.22.0/ you will notice empty folders without files.

[cvgw]

This should be fixed now, can you give tag a1af057f997316bfb1c4d2d82719d78481a02a79 or debug-a1af057f997316bfb1c4d2d82719d78481a02a79 a try?

[drptbl]

@cvgw Thanks for looking in to it 🙏. Re-built from scratch using version above, still failing (it passes with success while building with docker run):

$ executor --cache=true --cache-ttl=9h --context $CI_PROJECT_DIR/kaniko --dockerfile $CI_PROJECT_DIR/kaniko/Dockerfile --destination "$IMAGE_NAME:$TAG-buster-kaniko"
00:22
 INFO[0000] Resolved base name us.gcr.io/.../docker-node:10.19.0-buster-kaniko to us.gcr.io/.../docker-node:10.19.0-buster-kaniko 
 INFO[0000] Resolved base name us.gcr.io/.../docker-node:10.19.0-buster-kaniko to us.gcr.io/.../docker-node:10.19.0-buster-kaniko 
 INFO[0000] Retrieving image manifest us.gcr.io/.../docker-node:10.19.0-buster-kaniko 
 INFO[0000] Retrieving image manifest us.gcr.io/.../docker-node:10.19.0-buster-kaniko 
 INFO[0000] Built cross stage deps: map[]                
 INFO[0000] Retrieving image manifest us.gcr.io/.../docker-node:10.19.0-buster-kaniko 
 INFO[0000] Retrieving image manifest us.gcr.io/.../docker-node:10.19.0-buster-kaniko 
 INFO[0000] cmd: USER                                    
 INFO[0000] Checking for cached layer us.gcr.io/.../docker-e2e/cache:e690c8dfc211bcadd3104f3a7f5792b9eda04a6de7692c5496a896f9d8dc3fba... 
 INFO[0000] No cached layer found for cmd RUN apt-get update 
 INFO[0000] cmd: USER                                    
 INFO[0000] Unpacking rootfs as cmd RUN apt-get update requires it. 
 INFO[0011] Taking snapshot of full filesystem...        
 INFO[0020] Resolving paths                              
 INFO[0022] USER root                                    
 INFO[0022] cmd: USER                                    
 INFO[0022] No files changed in this command, skipping snapshotting. 
 INFO[0022] RUN apt-get update                           
 INFO[0022] cmd: /bin/sh                                 
 INFO[0022] args: [-c apt-get update]                    
 error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: no such file or directory
 ERROR: Job failed: command terminated with exit code 1

Dockerfile: https://gist.github.com/drptbl/92b67ccffa2c14bd78b7100e9d777a24

[cvgw]

@drptbl which docker image are you using? Are you using one of the tags I suggested? I.E gcr.io/kaniko-project/exectutor:a1af057f997316bfb1c4d2d82719d78481a02a79 or gcr.io/kaniko-project/exectutor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79?

I tested your example Dockerfile with both of those images and it built successfully; perhaps I made an error in testing.

Edit: I see the Dockerfile you just linked is not the same one as the original example. I'll take a look at the new one when I have a chance.

[drptbl]

@cvgw I've linked full Dockerfile now and started thread with just short version of it :). I've tested it with gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79, everything was being built with new kaniko, base image too (it relies on official node:10-buster image, it's just re-built with kaniko before and used as base for e2e image, so everything is ran with help of kaniko). Sadly it didn't work. Before it was an error related to yarn, now it fails even faster on apt-get install stage.

[drptbl]

@cvgw could you please remove fixed-needs-verification label? Thanks 🤝.

[cvgw]

@drptbl can you provide instructions on how to build the base image in your latest example or share an example that depends on a public base image. Thanks

[drptbl]

@cvgw

1. Build image named docker-node:10.19.0-buster using kaniko, with following Dockerfile: https://gist.github.com/drptbl/3441afdf9b5821d34b8ba399c011d755

example command format:

executor --cache=true --cache-ttl=9h --context $CI_PROJECT_DIR/kaniko --dockerfile $CI_PROJECT_DIR/kaniko/Dockerfile --destination "$IMAGE_NAME:$TAG-buster"

2. Build image named docker-e2e:10.19.0-buster using kaniko, with following Dockerfile (it uses previous image as base, please edit): https://gist.github.com/drptbl/92b67ccffa2c14bd78b7100e9d777a24

That's it. You will get an error while building secondary image (docker-e2e):

$ executor --cache=true --cache-ttl=9h --context $CI_PROJECT_DIR/kaniko --dockerfile $CI_PROJECT_DIR/kaniko/Dockerfile --destination "$IMAGE_NAME:$TAG-buster"
00:23
 INFO[0000] Resolved base name us.gcr.io/edited/docker-node:10.19.0-buster to us.gcr.io/edited/docker-node:10.19.0-buster 
 INFO[0000] Resolved base name us.gcr.io/edited/docker-node:10.19.0-buster to us.gcr.io/edited/docker-node:10.19.0-buster 
 INFO[0000] Retrieving image manifest us.gcr.io/edited/docker-node:10.19.0-buster 
 INFO[0000] Retrieving image manifest us.gcr.io/edited/docker-node:10.19.0-buster 
 INFO[0000] Built cross stage deps: map[]                
 INFO[0000] Retrieving image manifest us.gcr.io/edited/docker-node:10.19.0-buster 
 INFO[0000] Retrieving image manifest us.gcr.io/edited/docker-node:10.19.0-buster 
 INFO[0001] cmd: USER                                    
 INFO[0001] Checking for cached layer us.gcr.io/edited/docker-e2e/cache:3275a5d30d41e3700c70ac09944d85b9e34291b57c7971839f7dcc3091ecba6b... 
 INFO[0001] No cached layer found for cmd RUN apt-get update 
 INFO[0001] cmd: USER                                    
 INFO[0001] Unpacking rootfs as cmd RUN apt-get update requires it. 
 INFO[0012] Taking snapshot of full filesystem...        
 INFO[0020] Resolving paths                              
 INFO[0022] USER root                                    
 INFO[0022] cmd: USER                                    
 INFO[0022] No files changed in this command, skipping snapshotting. 
 INFO[0022] RUN apt-get update                           
 INFO[0022] cmd: /bin/sh                                 
 INFO[0022] args: [-c apt-get update]                    
 error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: no such file or directory
 ERROR: Job failed: command terminated with exit code 1

As you can see I'm always clearing cache while trying things, so it's not a case.

[cvgw]

@drptbl I cannot repro with the information you've provided using any of the following images

• fb0cfd74649f88a942c0a77c85136b791a9c09d1
• debug-a1af057f997316bfb1c4d2d82719d78481a02a79
• a1af057f997316bfb1c4d2d82719d78481a02a79

Can you please include the exact docker command you are using to execute the build

Dockerfile base https://gist.github.com/drptbl/3441afdf9b5821d34b8ba399c011d755 Dockerfile https://gist.github.com/drptbl/92b67ccffa2c14bd78b7100e9d777a24

docker command

docker run \
  -v `pwd`:/workspace \
  $IMAGE
  --context dir:///workspace \
  --dockerfile Dockerfile \
  --destination $DESTINATION \
  --cache=true \
  -v info

[drptbl]

@cvgw It didn't work for both commands, with cache and without, tried both. Also didn't work with other combinations (tried adding --cleanup and --single-snapshot).

1. Build docker-node image with: (assuming that Dockerfile is in root directory) https://gist.github.com/drptbl/3441afdf9b5821d34b8ba399c011d755

   executor --cache=true --cache-ttl=9h --context ./ --dockerfile ./Dockerfile --destination "us.gcr.io/edited/docker-node:10.19.0-buster"

2. Build docker-e2e image with: (assuming that Dockerfile is in another root directory) https://gist.github.com/drptbl/92b67ccffa2c14bd78b7100e9d777a24

   executor --cache=true --cache-ttl=9h --context ./ --dockerfile ./Dockerfile --destination "us.gcr.io/edited/docker-e2e:10.19.0-buster"

   Keep in mind that it also doesn't work for me if I remove --cache and --cache-ttl.

I'm building it on latest stable Kubernetes (gke) through GitLab using kaniko docker image (gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79), some more details about machine:

 Running with gitlab-runner 12.7.0 (58272c27)
   on gke-gitlab-runner-57d548b8f-bxlw7 3HmdzdD_
Using Kubernetes namespace: gitlab-runner
00:00
 Using Kubernetes executor with image gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79 ...
Waiting for pod gitlab-runner/runner-3hmdzdd-project-12747902-concurrent-0dppww to be running, status is Pending
00:06
 Waiting for pod gitlab-runner/runner-3hmdzdd-project-12747902-concurrent-0dppww to be running, status is Pending
 Running on runner-3hmdzdd-project-12747902-concurrent-0dppww via gke-gitlab-runner-57d548b8f-bxlw7...
.............

GitLab config (this way my repo is mounted by default on root dir, using image directly without docker run):

build:kaniko:
  image: $KANIKO_IMAGE
  stage: build
  tags:
    - gke
  script:
    - executor --cache=true --cache-ttl=9h --context $CI_PROJECT_DIR/kaniko --dockerfile $CI_PROJECT_DIR/kaniko/Dockerfile --destination "$IMAGE_NAME:$TAG-buster-kaniko"
    - echo "============== BUILT IMAGES =============="
    - echo "1. $IMAGE_NAME:$TAG-buster-kaniko"
  cache: {}
  dependencies: []
  allow_failure: true

@edit adding -v info didn't seem to change log output: https://gist.github.com/drptbl/cbbb188f62364f331221e045aee28544

[cvgw]

@drptbl can you please try it using a local docker daemon rather than gitlab's runner

[drptbl]

@cvgw Tested on macOS locally and it worked fine. How to make it work on GitLab runners? Any ideas? 🙏

Some dependencies are missing? docker works fine :(.

Thanks for helping me out and digging in to it.

[cvgw]

Thanks for testing @drptbl. I've never used gitlab before but I'm setting an account up now to debug this.

[drptbl]

@cvgw I can see how much this tool has reduced ours CI build time, even if output builds aren't usable (yet). Happy to help for such a great project :). Ask me anything.

[cvgw]

@drptbl I wasn't able to repro this on Gitlab either; I'm guessing something is different about how I set things up.

My runner is installed on a GKE cluster I built the base image using docker and pushed it to gitlab registry

Dockerfile base https://gist.github.com/drptbl/3441afdf9b5821d34b8ba399c011d755 Dockerfile https://gist.github.com/drptbl/92b67ccffa2c14bd78b7100e9d777a24

.gitlab-ci.yml

build:kaniko:
  image: $KANIKO_IMAGE
  stage: build
  tags:
    - gke
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - executor --cache=true --cache-ttl=9h --context $CI_PROJECT_DIR/kaniko --dockerfile $CI_PROJECT_DIR/kaniko/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
    - echo "============== BUILT IMAGES =============="
    - echo "1. $IMAGE_NAME:$TAG-buster-kaniko"
  cache: {}
  dependencies: []
  allow_failure: true

kaniko image gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79

[drptbl]

@cvgw just to confirm, I'm building base image with kaniko and then e2e image with kaniko. You wrote that you built base with docker. Did you mean kaniko or docker? Maybe the case is that base has to be built with kaniko too?

Flow:

1. build base image (docker-node) with kaniko and push it to registry
2. reference it in docker-e2e dockerfile
3. build docker-e2e image with kaniko
^ all of these done on gitlab runners running on kubernetes (which is running on gke) through gitlab.com

[cvgw]

@cvgw just to confirm, I'm building base image with kaniko and then e2e image with kaniko. You wrote that you built base with docker. Did you mean kaniko or docker? Maybe the case is that base has to be built with kaniko too?

I indeed built my base image with docker; I'll try building it with kaniko and report back

[drptbl]

@cvgw tried v0.18.0-debug, same error.

[drptbl]

@cvgw tried v0.19.0-debug, same error.

[Frederik-Baetens]

I also have this issue on gitlab runners, with a dockerfile that uses npm to build a project and my /usr/src/app/dist/ went missing when building with kaniko. i switched back to dind for now.

My docker file:

FROM node:12.13-alpine AS builder
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build:raw

FROM node:12.13-alpine
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install --unsafe-perm --only=production
COPY --from=builder ./dist ./dist
ENV NODE_ENV production
# Need to be set by deployment
ENV PROVIDER ''
EXPOSE 3000
CMD ["npm", "run start:raw"]

Edit: my issue is probably related to #552 and #1206

[drptbl]

I also have this issue on gitlab runners, with a dockerfile that uses npm to build a project and my /usr/src/app/dist/ went missing when building with kaniko. i switched back to dind for now.

Yup, there is no other way. I hope it will be fixed soon, kaniko is a great project 🎉.

[jmuleiro]

Hello there! I know this issue is rather old, but as it is still open, I'm sharing my experience with it.

I'm running Kaniko in Kubernetes as a temporary pod.

The Dockerfile is rather large, but nothing seemed to help the fact that all files were deleted once Kaniko pushed the built image to the destination. My solution came from changing the executor's build options.

Pod args:

# ...
    args:
      - "--context=<context>"
      - "--dockerfile=./docker/Dockerfile-stage"
      - "--destination=<gcr-destination>"
      - "--ignore-path=./node_modules"
      - "--ignore-path=./k8s"
      - "--ignore-path=./envs"
      - "--ignore-path=./scripts"
      - "--snapshotMode=time"
      - "--cache=true"
      - "--cache-ttl=9h"
# ..

I used to have the single-snapshot option on - this seemed to be the issue somehow. I removed it and now it works as expected.

[mecampbellsoup]

Perhaps related to https://github.com/GoogleContainerTools/kaniko/issues/2065 ?