search

GoogleContainerTools / kaniko

Build Container Images In Kubernetes
Apache License 2.0
14.88k stars 1.44k forks source link

Caching does not work for multi-build stage Dockerfiles with COPY from= #1520

Closed psmit closed 1 year ago

psmit commented 3 years ago

Some background. I'm am trying to use cargo chef in order to cache Rust dependencies in my Dockerfiles. They have a suggested Dockerfile (see homepage), but the principles it is built on don't work in Kaniko.

Actual behavior When building a multi-stage Dockerfile, layers are not cached correctly. Looking to the Dockerfile below, the expensive "sleep 120" happens on any build with changed files, even though it should only happen when the number of files change (resulting in a different /recipe).

Expected behavior My expectation is that, like normal docker build, the layers after COPY --from=recipe_lanner /recipe / are cached correctly and are only dependent on the contents of /recipe

To Reproduce Steps to reproduce the behavior:

  1. Create a single file in the build context. E.g. echo "good morning" > greeting
  2. Build the Dockerfile below with cache=true. Observe that "RUN sleep 120" is executed
  3. Change the file echo "good afternoon" > greeting 4 Build the Dockerfile again with cache=true. Observe that a) /recipe is identical as before and b) layers are yet not cached and the "RUN sleep 120" is again executed.

Additional Information

FROM base as recipe_planner COPY . . RUN find / -type f | wc -l > recipe RUN cat recipe

FROM base as recipe_cooker COPY --from=recipe_planner /recipe / RUN sleep 120 RUN cat /recipe > /cooked

FROM base as real_work COPY --from=recipe_cooker /cooked / COPY . .


 - Build Context
   See above, use a single file
 - Kaniko Image (fully qualified with digest): gcr.io/kaniko-project/executor:v1.3.0 / sha256:b9eec410fa32cd77cdb7685c70f86a96debb8b087e77e63d7fe37eaadb178709

 **Triage Notes for the Maintainers**
 <!-- 🎉🎉🎉 Thank you for an opening an issue !!! 🎉🎉🎉
We are doing our best to get to this. Please help us by helping us prioritize your issue by filling the section below -->

 | **Description** | **Yes/No** |
 |----------------|---------------|
 | Please check if this a new feature you are proposing        | <ul><li>- [ ] </li></ul>|
 | Please check if the build works in docker but not in kaniko | <ul><li>- [x] </li></ul>| 
 | Please check if this error is seen when you use `--cache` flag | <ul><li>- [x] </li></ul>|
 | Please check if your dockerfile is a multistage dockerfile | <ul><li>- [x] </li></ul>|
radupopa2010 commented 3 years ago

I had problems with caching and decided to test your docker file. In my case I got a unsupported status code 504; body: <html><body><h1>504 Gateway Time-out</h1> In my case it seems that the sleep cmd was indeed cached.

INFO[0002] Resolved base name busybox to base           
INFO[0002] Resolved base name base to recipe_planner    
INFO[0002] Resolved base name base to recipe_cooker     
INFO[0002] Resolved base name base to real_work         
INFO[0002] Retrieving image manifest busybox            
INFO[0002] Retrieving image busybox                     
INFO[0003] Retrieving image manifest busybox            
INFO[0003] Retrieving image busybox                     
INFO[0006] Built cross stage deps: map[1:[/recipe] 2:[/cooked]] 
INFO[0006] Retrieving image manifest busybox            
INFO[0006] Retrieving image busybox                     
INFO[0008] Retrieving image manifest busybox            
INFO[0008] Retrieving image busybox                     
INFO[0010] Executing 0 build triggers                   
INFO[0010] Skipping unpacking as no commands require it. 
INFO[0010] WORKDIR /                                    
INFO[0010] cmd: workdir                                 
INFO[0010] Changed working directory to /               
INFO[0010] No files changed in this command, skipping snapshotting. 
INFO[0010] Storing source image from stage 0 at path /kaniko/stages/0 
INFO[0010] Deleting filesystem...                       
INFO[0010] Base image from previous stage 0 found, using saved tar at path /kaniko/stages/0 
INFO[0010] Executing 0 build triggers                   
INFO[0010] Checking for cached layer ORG_NAME/REPO_NAME:acccfcd29a9464640715c197ba311943dcabda2d14f9ba5e64e88d7be0141667... 
INFO[0012] Using caching version of cmd: RUN find / -type f | wc -l > recipe 
INFO[0012] Checking for cached layer ORG_NAME/REPO_NAME:6a6c36f7d5eaa4967672edc365f5dbfe1666399cd29c1b1341702c6a4e97db78... 
INFO[0014] Using caching version of cmd: RUN cat recipe 
INFO[0014] Unpacking rootfs as cmd COPY . . requires it. 
INFO[0015] COPY . .                                     
INFO[0015] Taking snapshot of files...                  
INFO[0015] RUN find / -type f | wc -l > recipe          
INFO[0015] Found cached layer, extracting to filesystem 
INFO[0015] RUN cat recipe                               
INFO[0015] Found cached layer, extracting to filesystem 
INFO[0015] Saving file recipe for later use             
INFO[0015] Deleting filesystem...                       
INFO[0015] Base image from previous stage 0 found, using saved tar at path /kaniko/stages/0 
INFO[0015] Executing 0 build triggers                   
INFO[0015] Checking for cached layer ORG_NAME/REPO_NAME:8380aa8aac3450e80fcda720c501bd640b0ba5adde7cfc2427ead9da26220547... 
INFO[0018] Using caching version of cmd: RUN sleep 20   
INFO[0018] Checking for cached layer ORG_NAME/REPO_NAME:6db99b70d505e22e75ddca7408de8b98b8198b8f2a4cf2cc8a4d68a1b384a888... 
INFO[0021] Using caching version of cmd: RUN cat /recipe > /cooked 
INFO[0021] Unpacking rootfs as cmd COPY --from=recipe_planner /recipe / requires it. 
INFO[0021] COPY --from=recipe_planner /recipe /         
INFO[0021] Taking snapshot of files...                  
INFO[0021] RUN sleep 20                                 
INFO[0021] Found cached layer, extracting to filesystem 
INFO[0021] RUN cat /recipe > /cooked                    
INFO[0021] Found cached layer, extracting to filesystem 
INFO[0022] Saving file cooked for later use             
INFO[0022] Deleting filesystem...                       
INFO[0022] Base image from previous stage 0 found, using saved tar at path /kaniko/stages/0 
INFO[0022] Executing 0 build triggers                   
INFO[0022] Unpacking rootfs as cmd COPY --from=recipe_cooker /cooked / requires it. 
INFO[0022] COPY --from=recipe_cooker /cooked /          
INFO[0022] Taking snapshot of files...                  
INFO[0022] COPY . .                                     
INFO[0022] Taking snapshot of files...                  

real    0m27,505s
user    0m0,053s
sys 0m0,042s
aaron-prindle commented 1 year ago

This should be fixed now with: https://github.com/GoogleContainerTools/kaniko/pull/2559

Closing