search

docker / hub-feedback

Feedback and bug reports for the Docker Hub
https://hub.docker.com
233 stars 48 forks source link

Consecutive "docker build" calls resulting on distinct layer hashes due "/etc" directory changes #2274

Closed dantonformaai closed 1 year ago

dantonformaai commented 1 year ago

Problem description

Given this simplified Dockerfile:

FROM bash:4.4
RUN echo 1  # You might run this layer with distinct echo values in order to bypass layer cache

After running docker build twice and checking the hash of last layer for both created images using docker image inspect we can see that hashes don't match.

Checking the tar of both last layers we can see a single entry for /etc directory. I believe /etc has been changed by docker build due deletion/creation of following files by docker build after RUN|COPY|... finishes:

/etc/hostname
/etc/resolv.conf
/etc/hosts
/etc/mtab

Executing RUN touch -t 197001010000 /etc is not able to fix the issue since /etc is changed after RUN command finishes.

Recreating images after execute docker system prune generates a whole new list of hashes for layers below FROM. When sending those recreated images to Amazon Elastic Container Registry for example, all layers below FROM will be send again due new hash despite of content be the same except by attributes of /etc. It causes waiting, bandwidth consumption and increasing storage usage.

docker info output

Docker Info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.2)
  compose: Docker Compose (Docker Inc., v2.6.0)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 30
  Running: 1
  Paused: 0
  Stopped: 29
 Images: 151
 Server Version: 20.10.16
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
 runc version: v1.1.1-0-g52de29d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.104-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 11.68GiB
 Name: docker-desktop
 ID: 24JJ:SHUB:FFFH:V5P4:JH7F:GKDL:44GP:NENH:WOG5:APOA:QPAP:ZTV4
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

Task List

binman-docker commented 1 year ago

Hi @dantonformaai - I'm going to close this issue because it's not related to Docker Hub. You might try opening an issue in https://github.com/moby/buildkit

That said, you'll never have separate builds with the same hash due to timestamps. You may find researching "reproducible builds" to help if this is a necessity for you.