Auth error when using a shared buildkit with different docker registry access levels

[sindreij]

Hi

We have a buildkit-instance in a kubernetes cluster that is used by multiple jobs. Each of these jobs have only access to pull and push docker images to a specific "directory" in our shared docker registry.

If we build IMAGE_A and then IMAGE_B after, we sometime get the following error:

$ buildctl --debug --addr=[..] build --progress=plain --frontend=dockerfile.v0 --opt filename=Dockerfile --opt build-arg:BUILD_DATE_USED_FOR_CACHE_BUSTING=2023-01-16T14:18:22.017751304+00:00 --metadata-file=metadata.json --local context=. --local dockerfile=. --output type=image,"name=registry.our-registry.no/image_b:a65bc2e1e0ea8a0104cf90a50a998816b445a0cd",push=true,registry.insecure=false
[..]
#8 exporting to image
#8 exporting layers done
#8 exporting manifest sha256:fe102642a1553897c82599a00df82df6f42b50c05bed0748f2b2be0c6a4a1449 0.0s done
#8 exporting config sha256:c1a8fa1deb8405196deaec568d843a58299067d573c26d64552b699df2fdf3fc done
#8 pushing layers
time="2023-01-16T14:18:22Z" level=debug msg="stopping session" spanID=1578acaf0873e43f traceID=b63a546709af3c3720de02aabcf4fcf3
#8 pushing layers 0.1s done
#8 ERROR: failed to push registry.our-registry.no/image_b:a65bc2e1e0ea8a0104cf90a50a998816b445a0cd: server message: insufficient_scope: authorization failed

#9 [auth] image_a:pull image_b:pull,push token for registry.our-registry.no
#9 DONE 0.0s

#10 [auth] image_a:pull oklvws9jhedl/testapp/main:pull,push token for registry.headless-operations.no
#10 DONE 0.0s

However there is no referalls to IMAGE_A in the Dockerfile of IMAGE_B. So it's weird that building IMAGE_B fails because it cannot pull IMAGE_A.

[tonistiigi]

Is it possible for you to set up some runnable reproducer case so we can debug this?

[mouchar]

Hi, I'm also affected by this error. We're running buildkit in k8s and using it in our Gitlab CI pipeline. We have ~/docker/config.json containing docker.io credentials (pull-only, just to avoid rate limits), and at the start of every build job, we do docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY. The CI_REGISTRY_PASSWORD is short-lived and is valid only during the CI job run. Builder is shared among multiple Gitlab projects.

Buildkitd runs with the --debug flag so we have logs including a stack trace at the end of failed build session.

Occasionally, the builder fails to push, and I need to restart buildkitd pod to make it work again.

Logs from buildkitd are available here (pastebin).

EDIT: The following redactions were made in the log:

• COMPANY - name of Gitlab Organization
• IMAGE_BUILT - the last part of the image tag being just built (the full name is COMPANY/IMAGE_BUILT:sometag)
• OTHER_IMAGE - the last part of other image used to mount layers from (the full name is COMPANY/OTHER_IMAGE:sometag)

[mouchar]

@tonistiigi Although I was not able to set up a reproducer, I attached daemon debug logs from a build session when the error happened. See my previous comment. Did you find them useful? Is there anything that points to a specific part of code causing this issue?

[jmsariron]

Also affected by this issue. Dis someone get a way to fix this?

[sbidoul]

I'm also hitting this. Same scenario as https://github.com/moby/buildkit/issues/3509#issuecomment-1438359514

In the log, it ends with

#38 [auth] imagea:pull,push imageb:pull token for registry.my.domain`
#38 DONE 0.0s
#35 exporting to image
#35 pushing layers 0.8s done
#35 ERROR: failed to push registry.my.domain/imagea:0cede31a8b04cda834b3eaa32eec94e6751644b9: server message: insufficient_scope: authorization failed

imageb in the [auth] line is from another unrelated build.

So it seems it uses credentials of another build to push imagea to the registry.

Running buildctl prune on the buildkitd machine resolves the issue.

[jmsariron]

For now, I ended up building the image as a tarball and pushing it with crane, it's not a solution by any means, but it's better than nothing...

[tonistiigi]

@sbidoul What version of buildkit?

If you have 2 sets of credentials for the same registry/repo how do you even define this in Docker CLI config?

[sbidoul]

What version of buildkit?

@tonistiigi buildkit our persistent server pods run the v0.13.1 image. I think our client buildctl are still 0.12.4 (I need to check).

If you have 2 sets of credentials for the same registry/repo how do you even define this in Docker CLI config?

We configure credentials in ~/.docker/config.json in different client pods which are transient and run for one set of credentials at a time. Actually with the GitLab container registry you get per project credentials by default in GitLab CI jobs..

[jmsariron]

@sbidoul What version of buildkit?

If you have 2 sets of credentials for the same registry/repo how do you even define this in Docker CLI config?

With GitLab, you have a "Registry" per repository, all these under the same domain. But access to that registry is limited to users that have access to that repository.

When using a GitLab pipeline inside a repository, GitLab provides a Token inside the pipeline with acess to it's own registry, and the correct thing would be to use that token to auth and push the image.

I've seen that for example, Portainer allows to create auth configurations not only based on the registry domain, but also subpaths. Something like that would be useful here.

[minhnnhat]

I have exactly the same issue with @mouchar. Shared buildkit on k8s cluster, and Gitlab ci job to login and build using buildctl cli. Here is my gitlab-ci.yml file:

...
before_script:
    - |
      auth=$(echo -n "$CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD" | base64)
      echo "{\"auths\": {\"$CI_REGISTRY\": {\"auth\": \"$auth\"}}}" > ~/.docker/config.json
script:
    - |
      ...
      DOCKER_PUSH_TARGETS="$CI_REGISTRY_IMAGE:$CI_PIPELINE_ID,$CI_REGISTRY_IMAGE/${CI_PROJECT_NAME}${IMAGE_NAME_SUFFIX}:$CI_PIPELINE_ID"
      buildctl build \
      --frontend dockerfile.v0 \
      --local context=$CI_PROJECT_DIR \
      --local dockerfile=$CI_PROJECT_DIR \
      --output type=image,\"name=$DOCKER_PUSH_TARGETS\",push=true