Multistage builds doesn't always bust the cache correctly

Actual behavior When copying files from an other stage and then chowning the files, the chown command will be cached even though the files being copied have change, causing the cache to overwrite all changes.

Expected behavior I would expect the cache to be busted after the COPY-command if the files have changed.

To Reproduce Steps to reproduce the behavior:

Build the Dockerfile provided in this ticket with kaniko, and run:

$ docker run -it --rm gcr.io/project-xyz/app:hash cat myfile
VALUE

change the content of myfile to anything else and rerun step 1. It will still return VALUE

Additional Information The Dockerfile:

FROM alpine:3.10 AS builder

WORKDIR /source
COPY . .

FROM alpine:3.10

WORKDIR /target
COPY --from=builder /source .

RUN chown -R 1000:1000 .

The file beeing copied: myfile (content doesn't matter)

VALUE

Kaniko Image (fully qualified with digest)

Pulling image: gcr.io/kaniko-project/executor:latest
latest: Pulling from kaniko-project/executor
Digest: sha256:fb27a2ca60b0943d124100abb66e937bfcf81821d2c79eb7813b7e74637958d0
Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest

Output from build log (step 1)

Pulling image: gcr.io/kaniko-project/executor:latest
latest: Pulling from kaniko-project/executor
Digest: sha256:fb27a2ca60b0943d124100abb66e937bfcf81821d2c79eb7813b7e74637958d0
Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest
gcr.io/kaniko-project/executor:latest
INFO[0000] Resolved base name alpine:3.10 to alpine:3.10 
INFO[0000] Resolved base name alpine:3.10 to alpine:3.10 
INFO[0000] Resolved base name alpine:3.10 to alpine:3.10 
INFO[0000] Resolved base name alpine:3.10 to alpine:3.10 
INFO[0000] Downloading base image alpine:3.10           
INFO[0000] Error while retrieving image from cache: getting file info: stat /cache/sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a: no such file or directory 
INFO[0000] Downloading base image alpine:3.10           
INFO[0001] Downloading base image alpine:3.10           
INFO[0001] Error while retrieving image from cache: getting file info: stat /cache/sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a: no such file or directory 
INFO[0001] Downloading base image alpine:3.10           
INFO[0001] Built cross stage deps: map[0:[/source]]     
INFO[0001] Downloading base image alpine:3.10           
INFO[0001] Error while retrieving image from cache: getting file info: stat /cache/sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a: no such file or directory 
INFO[0001] Downloading base image alpine:3.10           
INFO[0002] Using files from context: [/workspace]       
INFO[0002] Taking snapshot of full filesystem...        
INFO[0003] WORKDIR /source                              
INFO[0003] cmd: workdir                                 
INFO[0003] Changed working directory to /source         
INFO[0003] Creating directory /source                   
INFO[0003] Taking snapshot of files...                  
INFO[0003] Using files from context: [/workspace]       
INFO[0003] COPY . .                                     
INFO[0003] Taking snapshot of files...                  
INFO[0003] Saving file /source for later use.           
INFO[0003] Deleting filesystem...                       
INFO[0003] Downloading base image alpine:3.10           
INFO[0003] Error while retrieving image from cache: getting file info: stat /cache/sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a: no such file or directory 
INFO[0003] Downloading base image alpine:3.10           
INFO[0003] Checking for cached layer gcr.io/project-xyz/app/cache:ae01cc20442f41c01abf119c027a6042dde6d5f9106ed9946fb8a73aa94419e9... 
INFO[0003] No cached layer found for cmd RUN chown -R 1000:1000 . 
INFO[0003] Unpacking rootfs as cmd RUN chown -R 1000:1000 . requires it. 
INFO[0004] Taking snapshot of full filesystem...        
INFO[0004] WORKDIR /target                              
INFO[0004] cmd: workdir                                 
INFO[0004] Changed working directory to /target         
INFO[0004] Creating directory /target                   
INFO[0004] Taking snapshot of files...                  
INFO[0004] COPY --from=builder /source .                
INFO[0004] Taking snapshot of files...                  
INFO[0004] RUN chown -R 1000:1000 .                     
INFO[0004] cmd: /bin/sh                                 
INFO[0004] args: [-c chown -R 1000:1000 .]              
INFO[0004] Taking snapshot of full filesystem...        
INFO[0004] Pushing layer gcr.io/project-xyz/app/cache:ae01cc20442f41c01abf119c027a6042dde6d5f9106ed9946fb8a73aa94419e9 to cache now 
PUSH
DONE

Output from build log (step 2)

Pulling image: gcr.io/kaniko-project/executor:latest
latest: Pulling from kaniko-project/executor
Digest: sha256:fb27a2ca60b0943d124100abb66e937bfcf81821d2c79eb7813b7e74637958d0
Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest
gcr.io/kaniko-project/executor:latest
INFO[0000] Resolved base name alpine:3.10 to alpine:3.10 
INFO[0000] Resolved base name alpine:3.10 to alpine:3.10 
INFO[0000] Resolved base name alpine:3.10 to alpine:3.10 
INFO[0000] Resolved base name alpine:3.10 to alpine:3.10 
INFO[0000] Downloading base image alpine:3.10           
INFO[0000] Error while retrieving image from cache: getting file info: stat /cache/sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a: no such file or directory 
INFO[0000] Downloading base image alpine:3.10           
INFO[0001] Downloading base image alpine:3.10           
INFO[0001] Error while retrieving image from cache: getting file info: stat /cache/sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a: no such file or directory 
INFO[0001] Downloading base image alpine:3.10           
INFO[0002] Built cross stage deps: map[0:[/source]]     
INFO[0002] Downloading base image alpine:3.10           
INFO[0002] Error while retrieving image from cache: getting file info: stat /cache/sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a: no such file or directory 
INFO[0002] Downloading base image alpine:3.10           
INFO[0002] Using files from context: [/workspace]       
INFO[0002] Taking snapshot of full filesystem...        
INFO[0003] WORKDIR /source                              
INFO[0003] cmd: workdir                                 
INFO[0003] Changed working directory to /source         
INFO[0003] Creating directory /source                   
INFO[0003] Taking snapshot of files...                  
INFO[0003] Using files from context: [/workspace]       
INFO[0003] COPY . .                                     
INFO[0003] Taking snapshot of files...                  
INFO[0003] Saving file /source for later use.           
INFO[0003] Deleting filesystem...                       
INFO[0003] Downloading base image alpine:3.10           
INFO[0003] Error while retrieving image from cache: getting file info: stat /cache/sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a: no such file or directory 
INFO[0003] Downloading base image alpine:3.10           
INFO[0003] Checking for cached layer gcr.io/project-xyz/app/cache:ae01cc20442f41c01abf119c027a6042dde6d5f9106ed9946fb8a73aa94419e9... 
INFO[0003] Using caching version of cmd: RUN chown -R 1000:1000 . 
INFO[0003] Skipping unpacking as no commands require it. 
INFO[0003] Taking snapshot of full filesystem...        
INFO[0003] WORKDIR /target                              
INFO[0003] cmd: workdir                                 
INFO[0003] Changed working directory to /target         
INFO[0003] Creating directory /target                   
INFO[0003] Taking snapshot of files...                  
INFO[0004] COPY --from=builder /source .                
INFO[0004] Taking snapshot of files...                  
INFO[0004] RUN chown -R 1000:1000 .                     
INFO[0004] Found cached layer, extracting to filesystem 
INFO[0004] Taking snapshot of files...                  
PUSH
DONE

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [x]
Please check if this error is seen when you use `--cache` flag	- [x]
Please check if your dockerfile is a multistage dockerfile	- [x]

GoogleContainerTools / kaniko

Multistage builds doesn't always bust the cache correctly #845