search

moby / buildkit

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
https://github.com/moby/moby/issues/34227
Apache License 2.0
8.19k stars 1.16k forks source link

Inconsistent cache behavior from CI and local environment (or from env to env) #2010

Open paolomainardi opened 3 years ago

paolomainardi commented 3 years ago

Scenario: Building with docker-compose native build and inline cache active

Test-case: https://github.com/paolomainardi/buildkit-cache-testcase

Description::

As the title suggests, i'm experiencing inconsistencies while building the very same image on a CI environment and other environments, to be more generic it seems that for some reasons the resulting image can be used as a cache layer just from the generating environment, in this case Github Actions.

As you can see here: https://github.com/paolomainardi/buildkit-cache-testcase/runs/2051488604 The image is built re-using the cache layers previously built and pushed to the docker registry.

This is the executed script:

        run: |
          docker system prune --all -f
          docker-compose version
          docker-compose build test
          docker push paolomainardi/buildkit-cache-testcase:master

Problem:

Standing the previous chapter, i would expect to clone this repo, run the very same commands and benefit of cached layers, well of course, that is not the case:

❯ make docker
docker system prune --all -f || true
Deleted build cache objects:
y4snf8qfafm5r2mi24nilfxz6
8bfx2uhh51cp1enxdqh65lzjk
wj4qar606y589kibc07b0pjo0
v984426v7hf0u9856zzl4vfyw
n3czqza4ev11g6q7my8bptwic
9oe2drxcp5dy3cw00yjo2e19t
g1jnoaysax5ni4u1b8gm5obf7
2f3yjnm64hbx6weds0ti87mh8
4w7ajp48ldwlz42vm0mrxskuu
pz7mbcs6w6zu900b4e5tjswj7
hew39ciw0mzywxztfapesvpy3
k1tyjz18641063f4myadknrcd
i5uq3j12gunrb0agrkoor3wrp
vydfvvxwy7lrc78o7t7ok0601
krw17lj0gbv7asecwiofsu5o6
vbo3vfv0fcr3uw161guaeic9m
0oxs1igqo3t2lteilh0ijhy5h
r7wv3ol2jp9ri170ik09pipm7

Total reclaimed space: 2.284kB
docker rmi paolomainardi/buildkit-cache-testcase:master || true
Error: No such image: paolomainardi/buildkit-cache-testcase:master

DOCKER_BUILDKIT=1 BUILDKIT_PROGRESS=plain docker-compose build

Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Building test
#1 [internal] load build definition from Dockerfile
#1 sha256:4e0d82de6ef9d01492551c97db18f32a1b07deda624243c5e08bf5d20560264b
#1 DONE 0.0s

#1 [internal] load build definition from Dockerfile
#1 sha256:4e0d82de6ef9d01492551c97db18f32a1b07deda624243c5e08bf5d20560264b
#1 transferring dockerfile: 1.26kB done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:4e689bc9a2e3bcd362bc455b0479976fa12d84c405803487c330ada95a816f79
#2 transferring context: 64B done
#2 DONE 0.0s

#4 [internal] load metadata for docker.io/library/composer:1.10.13
#4 sha256:f89cd6e1b905d46af98e1b1d92d8403816220049e37e0e026fbf4dc559432495
#4 ...

#3 [internal] load metadata for ghcr.io/sparkfabrik/docker-php-drupal-nginx:1.17.6-alpine.d8
#3 sha256:7f61f660e342f820a32fc9d87f3c1a9589607ab44d104a756da387038bad6948
#3 DONE 0.9s

#4 [internal] load metadata for docker.io/library/composer:1.10.13
#4 sha256:f89cd6e1b905d46af98e1b1d92d8403816220049e37e0e026fbf4dc559432495
#4 DONE 1.3s

#11 [internal] load build context
#11 sha256:a1429e4d2e12fe2b58637a2b01e65aec8b2a5e388b5625d6b74dab386608f55e
#11 DONE 0.0s

#7 [base 1/7] FROM docker.io/library/composer:1.10.13@sha256:d88b67536ff08fb56ed5530927629e5c8cc717a1e073504eee646a9912fef445
#7 sha256:a6951ec5e7bf802c371ea3c42064acf7fe0afa3b4f4db318591e4770d51c3a70
#7 DONE 0.0s

#6 [nginx 1/3] FROM ghcr.io/sparkfabrik/docker-php-drupal-nginx:1.17.6-alpine.d8@sha256:9612e94913b18c8b5b5da947af097495b0e3a07c3079a85bfdf4d621afe719f3
#6 sha256:76e0608888c4d36db5931efb64ce9eef86f1273362a3e4a4ad17d570f75f3e1e
#6 DONE 0.0s

#5 importing cache manifest from paolomainardi/buildkit-cache-testcase:master
#5 sha256:6c288e16a80d678bf580a1696c6014d9eef84d4f4678edc6522b0a49567d7f70
#5 DONE 1.3s

#11 [internal] load build context
#11 sha256:a1429e4d2e12fe2b58637a2b01e65aec8b2a5e388b5625d6b74dab386608f55e
#11 transferring context: 1.22kB done
#11 DONE 0.0s

#7 [base 1/7] FROM docker.io/library/composer:1.10.13@sha256:d88b67536ff08fb56ed5530927629e5c8cc717a1e073504eee646a9912fef445
#7 sha256:a6951ec5e7bf802c371ea3c42064acf7fe0afa3b4f4db318591e4770d51c3a70
#7 resolve docker.io/library/composer:1.10.13@sha256:d88b67536ff08fb56ed5530927629e5c8cc717a1e073504eee646a9912fef445 done
#7 sha256:5821c81e84f77906e6ae8c9a2d016d4d635669ca595b12e949fcea518d6ed415 3.46kB / 3.46kB done
#7 sha256:ea5fa9a4426763da5ff6910361248ecdba4f82415b4b99c4041ecb478a292772 12.29kB / 12.29kB done
#7 sha256:b358d6dbbdff5c10cbe23c608b7ff9c6d1dd13331dd7dc7644b727ca5ea8e742 0B / 1.34MB 0.1s
#7 sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 0B / 2.80MB 0.1s
#7 sha256:d88b67536ff08fb56ed5530927629e5c8cc717a1e073504eee646a9912fef445 1.65kB / 1.65kB done
#7 sha256:0232d962484c6b9a1caa33a12f1beed4cb20996085056b4fb1591a9fd1d8c89f 0B / 1.23kB 0.1s
#7 sha256:b358d6dbbdff5c10cbe23c608b7ff9c6d1dd13331dd7dc7644b727ca5ea8e742 1.34MB / 1.34MB 0.2s done
#7 sha256:0c1d3ac04d2af7a9f0eee25eccc72e586400051054ffb4aff7cdf2f4ebc993e8 0B / 222B 0.2s
#7 sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 2.80MB / 2.80MB 0.3s done
#7 extracting sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c 0.1s done
#7 sha256:18ac31f486d7494f42b15b3b35c17c1852d8a61492953a31a115b19e79c23f39 0B / 10.32MB 0.3s
#7 sha256:0232d962484c6b9a1caa33a12f1beed4cb20996085056b4fb1591a9fd1d8c89f 1.23kB / 1.23kB 0.4s done
#7 extracting sha256:b358d6dbbdff5c10cbe23c608b7ff9c6d1dd13331dd7dc7644b727ca5ea8e742 0.1s done
#7 sha256:5b377704e5fe9667bfd97d424f93cd520ad812a73bad631636bdf6d0c0fe919a 0B / 499B 0.4s
#7 extracting sha256:0232d962484c6b9a1caa33a12f1beed4cb20996085056b4fb1591a9fd1d8c89f
#7 sha256:0c1d3ac04d2af7a9f0eee25eccc72e586400051054ffb4aff7cdf2f4ebc993e8 222B / 222B 0.4s done
#7 sha256:18ac31f486d7494f42b15b3b35c17c1852d8a61492953a31a115b19e79c23f39 9.44MB / 10.32MB 0.6s
#7 sha256:5b377704e5fe9667bfd97d424f93cd520ad812a73bad631636bdf6d0c0fe919a 499B / 499B 0.5s done
#7 extracting sha256:0232d962484c6b9a1caa33a12f1beed4cb20996085056b4fb1591a9fd1d8c89f done
#7 sha256:e5413552d081a039226d296e03a19bc9a8f96c74e883142e34fab30d0452eba4 0B / 14.78MB 0.6s
#7 extracting sha256:0c1d3ac04d2af7a9f0eee25eccc72e586400051054ffb4aff7cdf2f4ebc993e8 done
#7 sha256:4df5163823ca324dc377c1b55247e4a2444f70230eb61f9a020b8b507b0ab6ad 0B / 2.27kB 0.6s
#7 sha256:18ac31f486d7494f42b15b3b35c17c1852d8a61492953a31a115b19e79c23f39 10.32MB / 10.32MB 0.6s done
#7 sha256:e5413552d081a039226d296e03a19bc9a8f96c74e883142e34fab30d0452eba4 6.29MB / 14.78MB 0.7s
#7 sha256:4df5163823ca324dc377c1b55247e4a2444f70230eb61f9a020b8b507b0ab6ad 2.27kB / 2.27kB 0.7s done
#7 extracting sha256:18ac31f486d7494f42b15b3b35c17c1852d8a61492953a31a115b19e79c23f39 0.1s done
#7 extracting sha256:5b377704e5fe9667bfd97d424f93cd520ad812a73bad631636bdf6d0c0fe919a done
#7 sha256:52bb01a7d8f4dc372cc30b5f373d253aa00f14e20d98db5dad94ba51b558a887 0B / 17.07kB 0.7s
#7 sha256:a788e22aa32d056de1e67888a4da26b276e0daef03e79f8395597bf4c5fb7132 0B / 33.86MB 0.7s
#7 sha256:e5413552d081a039226d296e03a19bc9a8f96c74e883142e34fab30d0452eba4 10.67MB / 14.78MB 0.8s
#7 sha256:52bb01a7d8f4dc372cc30b5f373d253aa00f14e20d98db5dad94ba51b558a887 17.07kB / 17.07kB 0.8s done
#7 sha256:4448a8f62fc5dfaae3cbd161423b56eddc92343f822a6e3f2b448e9bfb23c764 0B / 189.47kB 0.8s
#7 extracting sha256:e5413552d081a039226d296e03a19bc9a8f96c74e883142e34fab30d0452eba4
#7 sha256:e5413552d081a039226d296e03a19bc9a8f96c74e883142e34fab30d0452eba4 14.78MB / 14.78MB 0.8s done
#7 sha256:a788e22aa32d056de1e67888a4da26b276e0daef03e79f8395597bf4c5fb7132 4.19MB / 33.86MB 0.9s
#7 sha256:b38d4abd006f8514cf22da6cb22badcb9b086a67e06dd08961af4a5b60df3ce7 0B / 259B 0.9s
#7 sha256:a788e22aa32d056de1e67888a4da26b276e0daef03e79f8395597bf4c5fb7132 14.68MB / 33.86MB 1.0s
#7 sha256:4448a8f62fc5dfaae3cbd161423b56eddc92343f822a6e3f2b448e9bfb23c764 189.47kB / 189.47kB 0.9s done
#7 sha256:bcad9424cc58d6c3275601946e3d803d9b54f1eeac6c5b5901b7268e3dd39067 0B / 509.18kB 1.0s
#7 sha256:a788e22aa32d056de1e67888a4da26b276e0daef03e79f8395597bf4c5fb7132 25.17MB / 33.86MB 1.1s
#7 sha256:b38d4abd006f8514cf22da6cb22badcb9b086a67e06dd08961af4a5b60df3ce7 259B / 259B 1.0s done
#7 sha256:bcad9424cc58d6c3275601946e3d803d9b54f1eeac6c5b5901b7268e3dd39067 509.18kB / 509.18kB 1.1s done
#7 sha256:e57136c2a1ce78c8c3fd49add0ee4491f721e1192553e12a244cab9816df5ffa 0B / 408B 1.1s
#7 sha256:aaa652d41bbcf99a0b4ddcef430a583c4217f80ff54e0e45b746264797a8b3ef 0B / 92B 1.1s
#7 sha256:a788e22aa32d056de1e67888a4da26b276e0daef03e79f8395597bf4c5fb7132 33.86MB / 33.86MB 1.2s done
#7 extracting sha256:e5413552d081a039226d296e03a19bc9a8f96c74e883142e34fab30d0452eba4 0.3s done
#7 sha256:e57136c2a1ce78c8c3fd49add0ee4491f721e1192553e12a244cab9816df5ffa 408B / 408B 1.2s done
#7 sha256:aaa652d41bbcf99a0b4ddcef430a583c4217f80ff54e0e45b746264797a8b3ef 92B / 92B 1.2s done
#7 extracting sha256:4df5163823ca324dc377c1b55247e4a2444f70230eb61f9a020b8b507b0ab6ad done
#7 extracting sha256:52bb01a7d8f4dc372cc30b5f373d253aa00f14e20d98db5dad94ba51b558a887 done
#7 extracting sha256:a788e22aa32d056de1e67888a4da26b276e0daef03e79f8395597bf4c5fb7132 0.1s
#7 extracting sha256:a788e22aa32d056de1e67888a4da26b276e0daef03e79f8395597bf4c5fb7132 0.5s done
#7 extracting sha256:4448a8f62fc5dfaae3cbd161423b56eddc92343f822a6e3f2b448e9bfb23c764 done
#7 extracting sha256:b38d4abd006f8514cf22da6cb22badcb9b086a67e06dd08961af4a5b60df3ce7 done
#7 extracting sha256:bcad9424cc58d6c3275601946e3d803d9b54f1eeac6c5b5901b7268e3dd39067
#7 extracting sha256:bcad9424cc58d6c3275601946e3d803d9b54f1eeac6c5b5901b7268e3dd39067 0.0s done
#7 extracting sha256:e57136c2a1ce78c8c3fd49add0ee4491f721e1192553e12a244cab9816df5ffa done
#7 extracting sha256:aaa652d41bbcf99a0b4ddcef430a583c4217f80ff54e0e45b746264797a8b3ef done
#7 DONE 2.3s

#8 [base 2/7] RUN apk add --no-cache patch nodejs npm python2 g++
#8 sha256:ad9a547188ed32aa4e964332a8735ed52fc6fcff4403b24548c003ade1e210e8
#8 0.207 fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
#8 0.351 fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
#8 0.531 (1/20) Upgrading musl (1.1.24-r9 -> 1.1.24-r10)
#8 0.544 (2/20) Installing libgcc (9.3.0-r2)
#8 0.550 (3/20) Installing libstdc++ (9.3.0-r2)
#8 0.566 (4/20) Installing binutils (2.34-r1)
#8 0.619 (5/20) Installing gmp (6.2.0-r0)
#8 0.628 (6/20) Installing isl (0.18-r0)
#8 0.644 (7/20) Installing libgomp (9.3.0-r2)
#8 0.650 (8/20) Installing libatomic (9.3.0-r2)
#8 0.655 (9/20) Installing libgphobos (9.3.0-r2)
#8 1.006 (10/20) Installing mpfr4 (4.0.2-r4)
#8 1.015 (11/20) Installing mpc1 (1.1.0-r1)
#8 1.021 (12/20) Installing gcc (9.3.0-r2)
^C#8 CANCELED
context canceled

My docker info:

❯ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 20.10.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.8.0-44-generic
 Operating System: Ubuntu 20.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.33GiB
 Name: spark-carbon-cto
 ID: 2VUG:W4M7:ONOJ:GABB:KWWA:KILS:KLAA:RJLE:MOCY:YGB2:L6H6:VYP3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

What i am doing or understood wrong here ? Should it supposed to be cached right ?

paolomainardi commented 3 years ago

Any feedback on this ? cc @tonistiigi sorry for mentioning you, i saw that you are the more involved one on the other issues.

hansbogert commented 3 years ago

I'm experiencing the very same, spent an evening already debugging this.

DragosPT commented 1 year ago

See https://github.com/moby/moby/issues/32816#issuecomment-910030001 There might be issues with different file masks - pipeline clone vs local clone