Problems with `--reproducible` (modified timestamp)

Actual behavior Using a minimal Dockerfile and that add an empty file and flag --reproducible results in different build hashes when built multiple times.

Expected behavior The same hash for the docker images.

To Reproduce Steps to reproduce the behavior:

Create Dockerfile

FROM ubuntu@sha256:7cfe75438fc77c9d7235ae502bf229b15ca86647ac01c844b272b56326d56184
COPY empty /

Create empty file
```
touch empty
```

Run kaniko build

docker run \
-v "$(pwd)"/context/:/workspace \
-v "$(pwd)"/config.json:/kaniko/.docker/config.json:ro \
gcr.io/kaniko-project/executor \
--reproducible \
--dockerfile Dockerfile \
--context dir:///workspace/ \
--destination <registry> \
--cache=false

Additional Information

Dockerfile: above Please provide either the Dockerfile you're trying to build or one that can reproduce this error.

Build Context

$ tree
── config.json
├── context
│   ├── Dockerfile
│   └── empty
└── run.sh (containing the command)

Kaniko Image: gcr.io/kaniko-project/executor latest 14c90714063c 4 weeks ago 63.1MB

EDIT 1: Running container-diff does not yield any differences

$ container-diff diff <IMG_0> <IMG_1> --type=history --type=file --type=size                                                                                                                                                                   

-----File-----

These entries have been added to <IMG_0> None

These entries have been deleted from <IMG_0> None

These entries have been changed between <IMG_0> and <IMG_1>: None

-----History-----

Docker history lines found only in <IMG_0>: None

Docker history lines found only in <IMG_1>: None

-----Size-----

Image size difference between <IMG_0> and <IMG_1>: None

Triage Notes for the Maintainers

EDIT 2: I used diffoscope to inspect the file system layers of the docker containers and found the following differences:

$ sudo diffoscope /var/lib/docker/overlay2/b961fc9f522d3e9bd9ab215e1292e5a6a5f461e61913bf47f041d49d7e3c3a07 /var/lib/docker/overlay2/fefa77080b6ef9f2bea5a425761f94d0f955c10768945f1626d211aba298ee34/
+++ /var/lib/docker/overlay2/fefa77080b6ef9f2bea5a425761f94d0f955c10768945f1626d211aba298ee34/
├── stat {}
│ @@ -1,8 +1,8 @@
│  
│    Size: 4096         Blocks: 8          IO Block: 4096   directory
│  Device: 259,3    Links: 4
│  Access: (0710/drwx--x---)  Uid: (    0/    root)   Gid: (    0/    root)
│  
│ -Modify: 2022-10-31 11:25:57.856492186 +0000
│ +Modify: 2022-10-31 11:29:02.389132829 +0000
│   --- /var/lib/docker/overlay2/b961fc9f522d3e9bd9ab215e1292e5a6a5f461e61913bf47f041d49d7e3c3a07/committed
├── +++ /var/lib/docker/overlay2/fefa77080b6ef9f2bea5a425761f94d0f955c10768945f1626d211aba298ee34/committed
│ ├── stat {}
│ │ @@ -1,8 +1,8 @@
│ │  
│ │    Size: 0          Blocks: 0          IO Block: 4096   regular empty file
│ │  Device: 259,3  Links: 1
│ │  Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
│ │  
│ │ -Modify: 2022-10-31 11:25:57.856492186 +0000
│ │ +Modify: 2022-10-31 11:41:02.903334450 +0000
│   --- /var/lib/docker/overlay2/b961fc9f522d3e9bd9ab215e1292e5a6a5f461e61913bf47f041d49d7e3c3a07/diff
├── +++ /var/lib/docker/overlay2/fefa77080b6ef9f2bea5a425761f94d0f955c10768945f1626d211aba298ee34/diff
│ ├── stat {}
│ │ @@ -1,8 +1,8 @@
│ │  
│ │    Size: 4096       Blocks: 8          IO Block: 4096   directory
│ │  Device: 259,3  Links: 2
│ │  Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
│ │  
│ │ +Modify: 2022-10-31 11:25:51.046396023 +0000
│ │ -Modify: 2022-10-31 11:25:57.829825142 +0000
│ ├── stat {}
│ │ @@ -1,8 +1,8 @@
│ │  
│ │    Size: 4096       Blocks: 8          IO Block: 4096   directory
│ │  Device: 259,3  Links: 2
│ │  Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
│ │  
│ │ -Modify: 2022-10-31 11:25:57.829825142 +0000
│ │ +Modify: 2022-10-31 11:25:51.046396023 +0000
│   --- /var/lib/docker/overlay2/b961fc9f522d3e9bd9ab215e1292e5a6a5f461e61913bf47f041d49d7e3c3a07/link
├── +++ /var/lib/docker/overlay2/fefa77080b6ef9f2bea5a425761f94d0f955c10768945f1626d211aba298ee34/link
│ @@ -1 +1 @@
│ -MJTPWNJJMXPGAGOR4AU5SGV3EU
│ +YY2LS2LHD5FII6IXAHIGIUHWCT
│ ├── stat {}
│ │ @@ -1,8 +1,8 @@
│ │  
│ │    Size: 26         Blocks: 8          IO Block: 4096   regular file
│ │  Device: 259,3  Links: 1
│ │  Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
│ │  
│ │ -Modify: 2022-10-31 11:25:57.806491480 +0000
│ │ +Modify: 2022-10-31 11:25:51.023062360 +0000
│   --- /var/lib/docker/overlay2/b961fc9f522d3e9bd9ab215e1292e5a6a5f461e61913bf47f041d49d7e3c3a07/lower
├── +++ /var/lib/docker/overlay2/fefa77080b6ef9f2bea5a425761f94d0f955c10768945f1626d211aba298ee34/lower
│ ├── stat {}
│ │ @@ -1,8 +1,8 @@
│ │  
│ │    Size: 57         Blocks: 8          IO Block: 4096   regular file
│ │  Device: 259,3  Links: 1
│ │  Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
│ │  
│ │ -Modify: 2022-10-31 11:25:57.806491480 +0000
│ │ +Modify: 2022-10-31 11:25:51.023062360 +0000
│   --- /var/lib/docker/overlay2/b961fc9f522d3e9bd9ab215e1292e5a6a5f461e61913bf47f041d49d7e3c3a07/work
├── +++ /var/lib/docker/overlay2/fefa77080b6ef9f2bea5a425761f94d0f955c10768945f1626d211aba298ee34/work
│ ├── stat {}
│ │ @@ -1,8 +1,8 @@
│ │  
│ │    Size: 4096       Blocks: 8          IO Block: 4096   directory
│ │  Device: 259,3  Links: 3
│ │  Access: (0700/drwx------)  Uid: (    0/    root)   Gid: (    0/    root)
│ │  
│ │ +Modify: 2022-10-31 11:25:51.023062360 +0000
│ │ -Modify: 2022-10-31 11:25:57.806491480 +0000
│ │   --- /var/lib/docker/overlay2/b961fc9f522d3e9bd9ab215e1292e5a6a5f461e61913bf47f041d49d7e3c3a07/work/work
│ ├── +++ /var/lib/docker/overlay2/fefa77080b6ef9f2bea5a425761f94d0f955c10768945f1626d211aba298ee34/work/work
│ │ ├── stat {}
│ │ │ @@ -1,8 +1,8 @@
│ │ │  
│ │ │    Size: 4096         Blocks: 8          IO Block: 4096   directory
│ │ │  Device: 259,3    Links: 2
│ │ │  Access: (0000/d---------)  Uid: (    0/    root)   Gid: (    0/    root)
│ │ │  
│ │ │ -Modify: 2022-10-31 11:25:57.806491480 +0000
│ │ │ +Modify: 2022-10-31 11:25:51.023062360 +0000
│ │ ├── stat {}
│ │ │ @@ -1,8 +1,8 @@
│ │ │  
│ │ │    Size: 4096         Blocks: 8          IO Block: 4096   directory
│ │ │  Device: 259,3    Links: 2
│ │ │  Access: (0000/d---------)  Uid: (    0/    root)   Gid: (    0/    root)
│ │ │  
│ │ │ +Modify: 2022-10-31 11:25:51.023062360 +0000
│ │ │ -Modify: 2022-10-31 11:25:57.806491480 +0000
│ ├── stat {}
│ │ @@ -1,8 +1,8 @@
│ │  
│ │    Size: 4096       Blocks: 8          IO Block: 4096   directory
│ │  Device: 259,3  Links: 3
│ │  Access: (0700/drwx------)  Uid: (    0/    root)   Gid: (    0/    root)
│ │  
│ │ -Modify: 2022-10-31 11:25:57.806491480 +0000
│ │ +Modify: 2022-10-31 11:25:51.023062360 +0000

It looks like they only differ in the modification time, which I guess should not happen when the --reproducible flag is passed, right?

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [ ]
Please check if this error is seen when you use `--cache` flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [ ]

GoogleContainerTools / kaniko

Problems with `--reproducible` (modified timestamp) #2304