Open dweomer opened 3 months ago
So, updating the system ca trust got me further, i.e.:
# runner startup tweaks
sudo ln -vs /etc/docker/certs.d/build.cache.svc/ca.crt /usr/local/share/ca-certificates/build-cache-svc.crt
sudo update-ca-certificates
But then I see this in the access logs for build.cache.svc:
10.42.165.22 - - [25/May/2024:16:14:08 +0000] "PUT /v2/my-project/my-image/manifests/my-tag HTTP/1.1" 201 0 "" "buildkit/v0.13"
2024-05-25T16:14:08.762990483Z time="2024-05-25T16:14:08.7627864Z" level=info msg="response completed" go.version=go1.20.8 http.request.contenttype="application/vnd.oci.image.index.v1+json" http.request.host=build.cache.svc http.request.id=f0f62fae-b358-4b4f-af1c-64d8f2ddbdc8 http.request.method=PUT http.request.remoteaddr="10.42.165.22:58088" http.request.uri="/v2/my-project/my-image/manifests/my-tag" http.request.useragent="buildkit/v0.13" http.response.duration=7.41057ms http.response.status=201 http.response.written=0
2024/05/25 16:14:08 http: TLS handshake error from 10.42.53.0:63128: tls: client didn't provide a certificate
2024/05/25 16:14:08 http: TLS handshake error from 10.42.53.0:45160: EOF
Progress! Now the issue is the client (buildx imagetools, I believe) is neglecting to send a client certificate when pushing to the private build cache registry. So, I tried these additional tweaks at runner startup:
# match default expectations of https://github.com/docker/setup-buildx-action
# sans a DOCKER_CONFIG env override (which fails due to attempted writes on a read-only volumeMount)
ln -vs /etc/docker/certs.d ~/.docker/
# match the translated buildkitd.toml registry ca/keypair entries shipped to buildkitd backends
sudo mkdir -p /etc/buildkit
sudo ln -vs /etc/docker/certs.d /etc/buildkit/certs
... but still no joy.
AFAIK I have correctly followed the mTLS setup for my build-cache registry, i.e.:
runner@ip-10-10-11-198:~$ ls -alF /etc/docker/certs.d/build.cache.svc/
total 0
dr-xr-x--- 3 root docker 140 May 25 16:14 ./
drwxr-xr-x 7 root root 122 May 25 16:15 ../
drwxr-xr-x 2 root root 100 May 25 16:14 ..2024_05_25_16_14_57.1642649139/
lrwxrwxrwx 1 root root 32 May 25 16:14 ..data -> ..2024_05_25_16_14_57.1642649139/
lrwxrwxrwx 1 root root 13 May 25 16:14 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root 18 May 25 16:14 client.cert -> ..data/client.cert
lrwxrwxrwx 1 root root 17 May 25 16:14 client.key -> ..data/client.key
hi @tonistiigi, i do not understand this edit:
our build.cache.svc
registry is secured via mTLS (with some dummy basic auth) which buildkitd backend(s) push to without issue but with the same config present on the client isn't recognized by buildx imagetools (some call into that go package, i forget) and so it doesnt expect the custom ca nor does it use the client certificate when negotiating an https connection. am i missing something? the "insecure registry" designation is a category error, from my perspective.
was digging into this further yesterday (because i am unreasonably angry that the awesome buildx imagetools fails like this) and it looks like a problem with the containerd resolver that imagetools is relying on (i even tried an /etc/containerd/certs.d/_default/hosts.toml
with no luck). something truncated in the setup, likely a gap between what ctr adds and imagetools expects.
also, docker push to this mTLS secured registry works fine via the tlscacert
, tlscert
, and tlskey
flags to docker but these are not picked up by buildx imagetools.
likely related issues:
Contributing guidelines
I've found a bug and checked that ...
Description
It looks as if the
imagetools.Opt
passed to theitpull := imagetools.New(imageopt)
line is lacking the necessaryRegistryConfig
to connect to a private registry signed by a CA that isn't included in the system ca-certificates BUT that individual builders are able to push to without issue (meaning, they are configured properly ... the build pulls from the private build cache registry successfully, honoring the private registry cache importer).Expected behaviour
Can successfully merge multi-platform manifests for blobs that have already been pushed to a private registry.
Actual behaviour
Cannot successfully merge multi-platform manifests for blobs that have already been pushed to a private registry.
Buildx version
github.com/docker/buildx v0.13.1+dweomer.1 5decc6f3cbed2286772783c6e89faf7d3b3bb985
Docker info
Builders list
Configuration
Build logs
Additional info
This is driven via github actions on a private runner, leveraging:
See also:
2462