Closed polarathene closed 1 year ago
It was nothing to do with caching, my local build with BuildKit wasn't using buildx
.
It appears to be specific to (Resolved: I used an outdated BuildKit from Docker Engine, buildx
with a builder using the docker-container
driver.buildx
had the one with actual --link
support)
Installed buildx
locally (via package manager, v0.9.1
), switched to docker-container
driver and attempted a build but got a weird failure to troubleshoot:
Workaround was to create the driver with network in host mode: --driver-opt network=host
, and now it builds (without the COPY --chown
option):
Now Dockerfile has --chown=clamav
added:
Observations:
--chown=root
will work. But I've not noticed it working for other common entries in /etc/passwd
(across the FROM
and COPY --from
images in Dockerfile, the buildx build container, and the host itself).COPY --chown
if I don't use the COPY --link
option.The linked official docs for those options don't seem to describe any reason that shouldn't work (and it does work without docker-container
driver). It's unclear why it works for the root user, but unable to reference anything else in /etc/passwd
and /etc/group
files in the container?
$ docker buildx create \
--driver docker-container \
--driver-opt network=host \
--name test-builder
$ docker buildx build \
--builder test-builder \
--cache-from type=local,src=/tmp/cache-clamav \
--cache-to type=local,dest=/tmp/cache-clamav,mode=max \
--output type=cacheonly \
.
Dockerfile
:
# syntax=docker.io/docker/dockerfile:1
# Line above required for BUILDKIT features (RUN with HereDoc, and COPY with --link)
# Installs the clamav packages, which updates /etc/passwd and /etc/group with clamav user:
FROM docker.io/debian:11-slim as stage-base
ARG DEBIAN_FRONTEND=noninteractive
# Inline shell script to run:
RUN <<EOF
apt-get -y update
apt-get -y upgrade
apt-get -y --no-install-recommends install clamav clamav-daemon ca-certificates
rm -rf /var/lib/apt/lists/*
EOF
# 230MB x2 in size added from /var/lib/clamav
FROM docker.io/debian:11-slim as stage-clamav
# Reference user + group lookups from stage-base files instead:
COPY --link --from=stage-base /etc/passwd /etc/group /etc/
# Copy over the latest DB updates from the official ClamAV image.
# Better than running `freshclam` (which would require 500MB+ extra memory during an image build)
COPY --link --from=docker.io/clamav/clamav:latest /var/lib/clamav /var/lib/clamav
# NOTE: buildx with the docker-container driver does not support mixing COPY with `--link` and `--chown`,
# Unless using numeric UID / GID values. Handling this on a separate layer thus doubles the cache size:
RUN chown -R clamav:clamav /var/lib/clamav
# The next stage can then COPY the clamav data, so that the built image doesn't incur the chown size penalty:
# NOTE: Build cache will increase by approx 460MB from the separate stage,
# Link here still incurs 230MB into build cache for this stage:
FROM stage-base as stage-final
COPY --link --from=stage-clamav /var/lib/clamav /var/lib/clamav
/tmp/cache-clamav
uses 744MB of disk. It should be at least half of that without the workaround above.chown -R
and changing the prior COPY --link
to COPY --chown
. I assume this is roughly the same and shouldn't lose any benefits with this approach regarding build cache and image build size?Closing as incompatibility is expected (could do with a better error output and documentation due to the backwards-compatibility situation silently ignoring --link
)
--chown
(since you can use a string name instead of numeric ID).# syntax=docker.io/docker/dockerfile:1
# Line above required for BUILDKIT features (RUN with HereDoc, and COPY with --link)
# Installs the clamav packages, which updates /etc/passwd and /etc/group with clamav user:
FROM docker.io/debian:11-slim as stage-base
ARG DEBIAN_FRONTEND=noninteractive
# Inline shell script to run:
RUN <<EOF
apt-get -y update
apt-get -y upgrade
apt-get -y --no-install-recommends install clamav clamav-daemon ca-certificates
rm -rf /var/lib/apt/lists/*
EOF
# Copy over the latest DB updates from the official ClamAV image.
# Better than running `freshclam` (which would require 500MB+ extra memory during an image build)
FROM docker.io/debian:11-slim as stage-clamav
# Reference user + group lookups from stage-base files instead:
COPY --link --from=stage-base /etc/passwd /etc/group /etc/
COPY --chown=clamav --from=docker.io/clamav/clamav:latest /var/lib/clamav /var/lib/clamav
# NOTE: Build cache will increase by approx 230MB from the separate stage
# `--link` below still adds 230MB into the build cache for this stage (thus doubling build cache size needed):
FROM stage-base as stage-final
COPY --link --from=stage-clamav /var/lib/clamav /var/lib/clamav
--chown
with numeric ID (add user in advance)# syntax=docker.io/docker/dockerfile:1
# Line above required for BUILDKIT features (RUN with HereDoc, and COPY with --link)
FROM docker.io/debian:11-slim as stage-base
RUN <<EOF
DEBIAN_FRONTEND=noninteractive
# Add the clamav user before installing the package, use an explicit UID to provide to COPY --chown
adduser --quiet --system --group --disabled-password --home /var/lib/clamav --no-create-home --uid 200 clamav
apt-get -y update
apt-get -y upgrade
apt-get -y --no-install-recommends install clamav clamav-daemon ca-certificates
rm -rf /var/lib/apt/lists/*
EOF
# Copy over the latest DB updates from the official ClamAV image.
# Better than running `freshclam` (which would require 500MB+ extra memory during an image build)
COPY --link --chown=200 --from=docker.io/clamav/clamav:latest /var/lib/clamav /var/lib/clamav
UPDATE 2: Resolved. This was a misunderstanding on my part. I didn't realize Docker Engine
20.10.22
from Dec 2022 still used BuildKitv0.8.4
from Apr 2021. buildxv0.9.1
was using BuildKitv0.11.0
instead where--link
is actually supported not silently ignored for backwards compatibility.UPDATE: See next message for a minimal reproduction. Not a cache issue, seems specific tobuildx
when using thedocker-container
driver.I also somehow missed an existing issue with the same error message.
Original report (invalid)
~~I am assuming this has something to do with caching we have used in CI (_via the Github Action `docker-build-push`_), as a local build does not fail like this.~~ ## Local This correctly uses existing layer cache up until the change for `COPY --link` adding `--chown=clamav`. Command run: ```bash DOCKER_BUILDKIT=1 docker build \ --tag 'mailserver-testing:ci' \ --build-arg VCS_VERSION=$(shell git rev-parse --short HEAD) \ --build-arg VCS_REVISION=$(shell cat VERSION) \ . ```Terminal output
``` [+] Building 45.8s (60/60) FINISHED => [internal] load build definition from Dockerfile 0.1s => => transferring dockerfile: 12.09kB 0.0s => [internal] load .dockerignore 0.2s => => transferring context: 34B 0.0s => resolve image config for docker.io/docker/dockerfile:1 1.7s => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81 0.0s => [internal] load .dockerignore 0.0s => [internal] load build definition from Dockerfile 0.0s => [internal] load metadata for docker.io/clamav/clamav:latest 0.0s => [internal] load metadata for docker.io/library/debian:11-slim 0.0s => [stage-base 1/49] FROM docker.io/library/debian:11-slim 0.0s => [internal] load build context 0.1s => => transferring context: 25.72kB 0.0s => CACHED FROM docker.io/clamav/clamav:latest 0.0s => CACHED [stage-base 2/49] COPY target/bin/sedfile /usr/local/bin/sedfile 0.0s => CACHED [stage-base 3/49] RUN chmod +x /usr/local/bin/sedfile 0.0s => CACHED [stage-base 4/49] COPY target/scripts/build/* /build/ 0.0s => CACHED [stage-base 5/49] COPY target/scripts/helpers/log.sh /usr/local/bin/helpers/log.sh 0.0s => CACHED [stage-base 6/49] RUN /bin/bash /build/packages.sh 0.0s => CACHED [stage-base 7/49] RUN <Full build log
``` /usr/bin/docker buildx build --cache-from type=local,src=/tmp/.buildx-cache --cache-to type=local,dest=/tmp/.buildx-cache-new,mode=max --iidfile /tmp/docker-build-push-r1dOWM/iidfile --output type=cacheonly --platform linux/amd64 --metadata-file /tmp/docker-build-push-r1dOWM/metadata-file . #1 [internal] load .dockerignore #1 transferring context: 59B done #1 DONE 0.0s #2 [internal] load build definition from Dockerfile #2 transferring dockerfile: 12.09kB done #2 DONE 0.0s #3 resolve image config for docker.io/docker/dockerfile:1 #3 ... #4 [auth] docker/dockerfile:pull token for registry-1.docker.io #4 DONE 0.0s #3 resolve image config for docker.io/docker/dockerfile:1 #3 DONE 0.5s #5 docker-image://docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc #5 resolve docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc 0.0s done #5 sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1[124](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:128)d72dadce04db8badbe7b08546 9.94MB / 9.94MB 0.1s done #5 extracting sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546 #5 extracting sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546 0.2s done #5 DONE 0.3s #6 [auth] clamav/clamav:pull token for registry-1.docker.io #6 DONE 0.0s #7 [auth] library/debian:pull token for registry-1.docker.io #7 DONE 0.0s #8 [internal] load metadata for docker.io/library/debian:11-slim #8 DONE 0.2s #9 [internal] load metadata for docker.io/clamav/clamav:latest #9 DONE 0.2s #10 [internal] load build context #10 DONE 0.0s #11 [stage-base 1/49] FROM docker.io/library/debian:11-slim@sha256:98d3b4b0cee264301eb1354e0b549323af2d0633e1c43375d0b25c01826b6790 #11 resolve docker.io/library/debian:11-slim@sha256:98d3b4b0cee264301eb1354e0b549323af2d0633e1c43375d0b25c01826b6790 0.0s done #11 DONE 0.0s #12 importing cache manifest from local:9656101879626942798 #12 DONE 0.0s #13 FROM docker.io/clamav/clamav:latest@sha256:314c46478306f1bbf3216e2a8ca4b3cb87ba5dd1e14fe4d43f0e3d13712a4af1 #13 resolve docker.io/clamav/clamav:latest@sha256:314c46478306f1bbf3216e2a8ca4b3cb87ba5dd1e14fe4d43f0e3d13712a4af1 #13 ... #10 [internal] load build context #10 transferring context: 295.46kB 0.0s done #10 DONE 0.1s #13 FROM docker.io/clamav/clamav:latest@sha256:314c46478306f1bbf3216e2a8ca4b3cb87ba5dd1e14fe4d43f0e3d13712a4af1 #13 resolve docker.io/clamav/clamav:latest@sha256:314c46478306f1bbf3216e2a8ca4b3cb87ba5dd1e14fe4d43f0e3d13712a4af1 0.0s done #13 sha256:24cb207b181c6103680fa838d6c19b2baa1869aee6afcd891ce46b7b174f7b26 1.26kB / 1.26kB 0.0s done #13 sha256:2d8c7c5ce39eaa53a3997135d3c16598f489f99655024514de17424d064b03e4 348B / 348B 0.0s done #13 sha256:68331520d622a7054a5e17fa38f2f5e25bf0680d93adfee677445f671ec4e3ba 1.66MB / 1.66MB 0.1s done #13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 7.34MB / 231.14MB 0.2s #13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 19.92MB / 231.14MB 0.3s #13 sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715 3.37MB / 3.37MB 0.1s done #13 sha256:cff7aab4088b8607efe477bdd4a9e0185bc8ed8339e171980897550e15390466 11.53MB / 78.00MB 0.2s #13 extracting sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715 #13 sha256:cff7aab4088b8607efe477bdd4a9e0185bc8ed8339e171980897550e15390466 30.41MB / 78.00MB 0.3s #13 extracting sha256:c158987b05517b6f2c5913f3acef1f2182a32345a304fe357e3ace5fadcad715 0.2s done #13 extracting sha256:68331520d622a7054a5e17fa38f2f5e25bf0680d93adfee677445f671ec4e3ba #13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 34.83MB / 231.14MB 0.5s #13 sha256:cff7aab4088b8607efe477bdd4a9e0185bc8ed8339e171980897550e15390466 47.63MB / 78.00MB 0.5s #13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 57.67MB / 231.14MB 0.6s #13 sha256:cff7aab4088b8607efe477bdd4a9e0185bc8ed8339e171980897550e15390466 59.77MB / 78.00MB 0.6s #13 sha256:cff7aab4088b8607efe477bdd4a9e0185bc8ed8339e171980897550e15390466 77.59MB / 78.00MB 0.8s #13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 83.96MB / 231.14MB 0.9s #13 sha256:cff7aab4088b8607efe477bdd4a9e0185bc8ed8339e171980897550e15390466 78.00MB / 78.00MB 0.9s done #13 extracting sha256:68331520d622a7054a5e17fa38f2f5e25bf0680d93adfee677445f671ec4e3ba 0.5s done #13 extracting sha256:cff7aab4088b8607efe477bdd4a9e0185bc8ed8339e171980897550e15390466 #13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 101.71MB / 231.14MB 1.1s #13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 114.29MB / 231.14MB 1.2s #13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 [131](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:135).76MB / 231.14MB 1.4s #13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 152.70MB / 231.14MB 1.5s #13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 170.92MB / 231.14MB 1.7s #13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 188.74MB / 231.14MB 1.8s #13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 214.96MB / 231.14MB 2.0s #13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 229.64MB / 231.14MB 2.1s #13 sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 231.14MB / 231.14MB 3.0s done #13 extracting sha256:cff7aab4088b8607efe477bdd4a9e0185bc8ed8339e171980897550e15390466 2.3s done #13 extracting sha256:2d8c7c5ce39eaa53a3997[135](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:139)d3c16598f489f99655024514de17424d064b03e4 done #13 extracting sha256:24cb207b181c6103680fa838d6c19b2baa1869aee6afcd891ce46b7b174f7b26 done #13 DONE 3.3s #13 FROM docker.io/clamav/clamav:latest@sha256:314c46478306f1bbf3216e2a8ca4b3cb87ba5dd1e14fe4d43f0e3d[137](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:141)12a4af1 #13 extracting sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d[183](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:187)2dd2e0f2f8cf9e33 #13 extracting sha256:06486e751a263a61f8582b3ef09ed4a80cdc2a3dbce6d1832dd2e0f2f8cf9e33 1.5s done #13 DONE 4.8s #14 [stage-base 3/49] RUN chmod +x /usr/local/bin/sedfile #14 CACHED #15 [stage-base 4/49] COPY target/scripts/build/* /build/ #15 CACHED #16 [stage-base 5/49] COPY target/scripts/helpers/log.sh /usr/local/bin/helpers/log.sh #16 CACHED #17 [stage-base 6/49] RUN /bin/bash /build/packages.sh #17 CACHED #18 [stage-base 2/49] COPY target/bin/sedfile /usr/local/bin/sedfile #18 CACHED #19 [stage-base 7/49] RUN <Run docker/build-push-action@v3.3.0
``` with: context: . platforms: linux/amd6[4](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:4) cache-from: type=local,src=/tmp/.buildx-cache cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max outputs: type=cacheonly load: false no-cache: false pull: false push: false github-token: *** ```Docker info (20.10.22)
``` /usr/bin/docker version Client: Version: 20.10.22+azure-1 API version: 1.41 Go version: go1.18.9 Git commit: 3a2c30b63ab20acfcc3f3550ea756a0561655a77 Built: Thu Dec 15 15:37:38 UTC 2022 OS/Arch: linux/amd64 Context: default Experimental: true Server: Engine: Version: 20.10.22+azure-1 API version: 1.41 (minimum version 1.12) Go version: go1.18.9 Git commit: 42c8b314993e5eb3cc2776da0bbe41d5eb4b707b Built: Thu Dec 15 22:17:04 2022 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.5.16+azure-1 GitCommit: 2e3140a0e09d[28](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:30)8a9086474752b4478aa0964e7c runc: Version: 1.1.4 GitCommit: 5fd4c4d144137e991c4acebb2146ab1483a97925 docker-init: Version: 0.19.0 GitCommit: /usr/bin/docker info Client: Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc., 0.9.1+azure-2) compose: Docker Compose (Docker Inc., 2.15.1+azure-1) Server: Containers: 1 Running: 1 Paused: 0 Stopped: 0 Images: 20 Server Version: 20.10.22+azure-1 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: false userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 2e3140a0e09d288a9086474752b4478aa0964e7c runc version: 5fd4c4d144137e991c4acebb2146ab1483a97925 init version: Security Options: apparmor seccomp Profile: default Kernel Version: 5.15.0-10[30](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:32)-azure Operating System: Ubuntu 20.04.5 LTS OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 6.781GiB Name: fv-az628-879 ID: J5HJ:OPXN:RYBU:[55](https://github.com/docker-mailserver/docker-mailserver/actions/runs/3945342546/jobs/6752151872#step:7:57)NP:DUWN:INRV:KGP3:GG3Y:CDQE:4K4D:LC24:LEZB Docker Root Dir: /var/lib/docker Debug Mode: false Username: githubactions Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false ```Buildx version (0.9.1)
``` /usr/bin/docker buildx version github.com/docker/buildx 0.9.1+azure-2 ed00243a0ce2a0aee75311b06e32d33b44729689 /usr/bin/docker buildx build --cache-from type=local,src=/tmp/.buildx-cache --cache-to type=local,dest=/tmp/.buildx-cache-new,mode=max --iidfile /tmp/docker-build-push-r1dOWM/iidfile --output type=cacheonly --platform linux/amd64 --metadata-file /tmp/docker-build-push-r1dOWM/metadata-file . ```