getting a 401 unauthorized when using a private HTTP insecure registry and docker-container drivers

[nirbenator]

we've been getting 401 errors since the release of 0.12.3 and its promotion to stable using an insecure HTTP registry

we've bypassed it by pinning the old 0.12.2 version when creating buildx builders/.

way to reproduce

docker buildx create --name default --platform linux/amd64 --config /buildx/buildkitd.toml --use 

cat <<EOF > Dockerfile
FROM alpine
RUN ls
EOF

docker buildx build . -t docker-registry:5000/ab --push

root@test-registry-agent-85f87fddf7-cfcb9:/# docker buildx build . -t docker-registry:5000/ab --push
[+] Building 1.6s (8/8) FINISHED
 => [internal] booting buildkit                                                                                         0.6s
 => => pulling image moby/buildkit:buildx-stable-1                                                                      0.2s
 => => creating container buildx_buildkit_buildkit-latest0                                                              0.5s
 => [internal] load build definition from Dockerfile                                                                    0.0s
 => => transferring dockerfile: 56B                                                                                     0.0s
 => [internal] load metadata for docker.io/library/alpine:latest                                                        0.4s
 => [internal] load .dockerignore                                                                                       0.0s
 => => transferring context: 2B                                                                                         0.0s
 => [1/2] FROM docker.io/library/alpine:latest@sha256:eece025e432126ce23f223450a0326fbebde39cdf496a85d8c016293fc851978  0.1s
 => => resolve docker.io/library/alpine:latest@sha256:eece025e432126ce23f223450a0326fbebde39cdf496a85d8c016293fc851978  0.0s
 => => sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa 3.40MB / 3.40MB                          0.1s
 => => extracting sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa                               0.1s
 => [2/2] RUN ls                                                                                                        0.1s
 => ERROR exporting to image                                                                                            0.1s
 => => exporting layers                                                                                                 0.1s
 => => exporting manifest sha256:ce2e890ceab4d1986adf07e1c1022eee9fcc0d8d1b396964e31ae5369af9647c                       0.0s
 => => exporting config sha256:a3b5720e265d6dfebb40b06061ebdf6af80530786f9420aad7af0eab7caed043                         0.0s
 => => pushing layers                                                                                                   0.0s
 => [auth] sharing credentials for docker-registry:5000                                                                 0.0s
------
 > exporting to image:
------
error: failed to solve: rpc error: code = Unknown desc = failed to push docker-registry:5000/ab: failed commit on ref "config-sha256:a3b5720e265d6dfebb40b06061ebdf6af80530786f9420aad7af0eab7caed043": unexpected status from PUT request to http://docker-registry:5000/v2/ab/blobs/uploads/54500b26-f64e-4b4c-8556-7b545b6bfa89?_state=5unnTSYL2WO7x2gsvfQIldK8PUKEQCvv4korEKZt6sJ7Ik5hbWUiOiJhYiIsIlVVSUQiOiI1NDUwMGIyNi1mNjRlLTRiNGMtODU1Ni03YjU0NWI2YmZhODkiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjMtMTAtMjRUMTE6NDA6MTQuOTE2Mjg3NzI0WiJ9&digest=sha256%3Aa3b5720e265d6dfebb40b06061ebdf6af80530786f9420aad7af0eab7caed043: 401 Unauthorized

toml config:

root@test-registry-agent-85f87fddf7-cfcb9:/# cat /buildx/buildkitd.toml
[registry."docker-registry:5000"]
  http = true
  insecure = true

way to work around:

docker buildx create --name buildkit-v0.12.2 --driver docker-container --driver-opt image=moby/buildkit:v0.12.2 --platform linux/amd64 --config /buildx/buildkitd.toml --use

cat <<EOF > Dockerfile
FROM alpine
RUN ls
EOF

docker buildx build . -t docker-registry:5000/ab --push
`

[crazy-max]

Also repro and discussed about it internally with @vvoland.

This is related to https://github.com/moby/buildkit/pull/4299 when pushing to an insecure registry: https://github.com/crazy-max/docker-build-push-action/actions/runs/6623261080/job/17990016733#step:11:377

ERROR: failed to solve: failed to push localhost:8081/test-docker-action/test-docker-action:test-e2e-harbor: failed commit on ref "layer-sha256:46fb11a1eeb257c79f02188ebaefafd1033faa92c06cc5dc46fd8e7973ce26d3": unexpected status from PUT request to http://localhost:8081/v2/test-docker-action/test-docker-action/blobs/uploads/aaef06c7-30d5-4494-8870-149ca532a248?_state=flumGm84a-y-BY9HQc4kW7X66UHHgAMoAEJXL2DklW97Ik5hbWUiOiJ0ZXN0LWRvY2tlci1hY3Rpb24vdGVzdC1kb2NrZXItYWN0aW9uIiwiVVVJRCI6ImFhZWYwNmM3LTMwZDUtNDQ5NC04ODcwLTE0OWNhNTMyYTI0OCIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyMy0xMC0yNFQwNjo0MDowMC45MjE2NDg2OFoifQ%3D%3D&digest=sha256%3A46fb11a1eeb257c79f02188ebaefafd1033faa92c06cc5dc46fd8e7973ce26d3: 401 Unauthorized

In BuildKit logs: https://github.com/docker/build-push-action/actions/runs/6545574556/job/17774356381#step:25:285

time="2023-10-17T10:03:52Z" level=debug msg="upload changed destination" digest="sha256:c7ce92c2dd2abe5f765108e6155701e9bb84322c4131a8fb40ab603d35ec8296" host="localhost:8081" mediatype=application/vnd.oci.image.layer.v1.tar+gzip scheme=http size=144

...we have this new line in the logs and seems related to the pusher logic in containerd where it strips Authorizer when scheme/host has changed: https://github.com/containerd/containerd/blob/18c9e7ec4c692cc00b7ac75d86d60685c270d48a/remotes/docker/pusher.go#L247-L255

[crazy-max]

https://github.com/containerd/containerd/pull/9294 should fix this issue.

[Or-Geva]

@nirbenator Thank you SO much for your workaround!!.

[crazy-max]

fixed by https://github.com/moby/buildkit/pull/4393 (https://github.com/containerd/containerd/pull/9299)

[martadinata666]

I'm assuming this fix happen by upgrading containerd to version 1.7.8? Yet Jammy repo follow 1.6 lineup https://download.docker.com/linux/ubuntu/dists/jammy/pool/stable/amd64/ is there easy way to upgrade? Dunno, if the this intended or not.

containerd/jammy-updates 1.7.2-0ubuntu1~22.04.1 amd64
  daemon to control runC

containerd.io/jammy,now 1.6.24-1 amd64 [installed]
  An open and reliable container runtime

[klit79]

still error on 0.12.4 i was losing my mind i didn't know image tag can be specify to 0.12.2

[crazy-max]

The fix has not been backported to 0.12. Keep you posted.

[ariwijayaikd]

@klit79 is it solved for you?