Image built by kaniko might have whiteout files that *seems* unnecessary

[ruiwen-zhao]

Actual behavior Image built by kaniko might have whiteout files that seems unnecessary. More details in the To Reproduce section below. This behavior caused image pulling error when the image is pulled and unpacked by containerd. See https://github.com/containerd/containerd/issues/4659

Expected behavior Image built by kaniko should not have whiteout files if the actual files or the directories are deleted on lower layers.

To Reproduce See https://github.com/containerd/containerd/issues/4659#issuecomment-1079487481 for details , but in general:

1. Build an image with the following Dockerfile

   
   FROM fluent/fluentd:v1.9.3-1.0

USER root

RUN apk add --no-cache --update --virtual .build-deps \ sudo build-base ruby-dev \ && sudo gem install \ fluent-plugin-nsq:0.0.6 \ fluent-plugin-kubernetes_metadata_filter:2.4.1 \ fluent-plugin-record-modifier:2.1.0 \ fluent-plugin-prometheus:1.7.3 \ fluent-plugin-throttle:0.0.4 \ fluent-plugin-multi-format-parser:1.0.0 \ && sudo gem sources --clear-all \ && apk del .build-deps \ && rm -rf /tmp/ /var/tmp/ /usr/lib/ruby/gems//cache/.gem

RUN gem install nsq-ruby:2.3.1 && gem uninstall -I nsq-ruby --version 1.7.0

2. Build the image with kaniko

tar -cf - Dockerfile | gzip -9 | docker run --interactive -v pwd/config.json:/kaniko/.docker/config.json:ro -v $(pwd):/workspace gcr.io/kaniko-project/executor:latest --context tar://stdin --destination qiutongs/kaniko-example:latest --force

3. Observe that top layer of the kaniko-built image is different than the image with same Dockerfile built by docker

Kaniko built:

root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.async-1.24.2.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.async-http-0.50.0.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.async-io-1.27.3.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.async-pool-0.2.0.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.bigdecimal-1.4.4.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.concurrent-ruby-1.1.6.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.console-1.8.2.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.cool.io-1.6.0.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.ext_monitor-0.1.2.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.fluentd-1.9.3.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.http_parser.rb-0.6.0.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.json-2.3.0.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.msgpack-1.3.3.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.nio4r-2.5.2.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.oj-3.8.1.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.protocol-hpack-1.4.2.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.protocol-http-0.13.1.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.protocol-http1-0.10.2.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.protocol-http2-0.10.4.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.serverengine-2.2.1.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.sigdump-0.2.4.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.strptime-0.2.3.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.timers-4.3.0.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.tzinfo-2.0.1.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.tzinfo-data-1.2019.3.gemspec root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.yajl-ruby-1.4.1.gemspec usr/lib/ruby/gems/2.5.0/gems/.wh.nsq-ruby-1.7.0 usr/lib/ruby/gems/2.5.0/specifications/.wh.nsq-ruby-1.7.0.gemspec / root/ root/.gem/ root/.gem/specs/ root/.gem/specs/api.rubygems.org%443/ root/.gem/specs/api.rubygems.org%443/quick/ root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/ root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/nsq-ruby-2.3.1.gemspec ...

Docker built:

root/ root/.gem/ root/.gem/specs/ root/.gem/specs/api.rubygems.org%443/ root/.gem/specs/api.rubygems.org%443/quick/ root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/ root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/nsq-ruby-2.3.1.gemspec ...

Note the whiteout files (`e.g. root/.gem/specs/api.rubygems.org%443/quick/Marshal.4.8/.wh.async-1.24.2.gemspec`) on the top layer of the kaniko-built image. The directory has actually been deleted in lower layers, so there is no need to have these whiteout files on the top layer. (And that's why the docker-built image doesn't have them)  

**Additional Information**
 - Dockerfile 

FROM fluent/fluentd:v1.9.3-1.0

USER root

RUN apk add --no-cache --update --virtual .build-deps \ sudo build-base ruby-dev \ && sudo gem install \ fluent-plugin-nsq:0.0.6 \ fluent-plugin-kubernetes_metadata_filter:2.4.1 \ fluent-plugin-record-modifier:2.1.0 \ fluent-plugin-prometheus:1.7.3 \ fluent-plugin-throttle:0.0.4 \ fluent-plugin-multi-format-parser:1.0.0 \ && sudo gem sources --clear-all \ && apk del .build-deps \ && rm -rf /tmp/ /var/tmp/ /usr/lib/ruby/gems//cache/.gem

RUN gem install nsq-ruby:2.3.1 && gem uninstall -I nsq-ruby --version 1.7.0

 - Build Context
   Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)
 - Kaniko Image (fully qualified with digest)

tar -cf - Dockerfile | gzip -9 | docker run --interactive -v pwd/config.json:/kaniko/.docker/config.json:ro -v $(pwd):/workspace gcr.io/kaniko-project/executor:latest --context tar://stdin --destination qiutongs/kaniko-example:latest --force

 **Triage Notes for the Maintainers**
 <!-- 🎉🎉🎉 Thank you for an opening an issue !!! 🎉🎉🎉
We are doing our best to get to this. Please help us by helping us prioritize your issue by filling the section below -->

 | **Description** | **Yes/No** |
 |----------------|---------------|
 | Please check if this a new feature you are proposing        | <ul><li>- [ ] </li></ul>|
 | Please check if the build works in docker but not in kaniko | <ul><li>- [ ] </li></ul>| 
 | Please check if this error is seen when you use `--cache` flag | <ul><li>- [ ] </li></ul>|
 | Please check if your dockerfile is a multistage dockerfile | <ul><li>- [ ] </li></ul>| 

[acouvreur]

We copy directly our build sources from the CI into the container, some of our files vanished in the final image. We replaced the default nginx landing page (index.html), but the default file was kept.

[gabyx]

This is likely to get fixed: in #2066 Which corrects some really strange bugs with deleting and adding files.

[aaron-prindle]

Can anyone in the thread here confirm if #2066 fixed this issue?

[ruiwen-zhao]

according to https://github.com/containerd/containerd/issues/4659#issuecomment-1206032086, this issue is fixed by #2066.

Closing.