Token endpoint regression with multiple `scope` parameters

[jonjohnsonjr]

It seems like Docker Hub recently updated their behavior in the token endpoint causing a regression in scope handling. There are a handful of affected clients, including containerd and go-containerregistry.

From https://docs.docker.com/registry/spec/auth/token/#requesting-a-token:

Query Parameters scope The resource in question, formatted as one of the space-delimited entries from the scope parameters from the WWW-Authenticate header shown above. This query parameter should be specified multiple times if there is more than one scope entry from the WWW-Authenticate header. The above example would be specified as: scope=repository:samalba/my-app:push. The scope field may be empty to request a refresh token without providing any resource permissions to the returned bearer token.

(Emphasis mine.)

@ekcasey encountered this issue:

2022/03/02 11:20:39 <-- 200 https://auth.docker.io/token?scope=repository%3Aekcasey%2Ftest1%3Apush%2Cpull&scope=repository%3Aekcasey%2Ftest%3Apull&scope=repository%3Aekcasey%2Ftest1%3Apull%2Cpush&service=registry.docker.io (84.440574ms) [body redacted: basic token response contains credentials]
2022/03/02 11:20:39 HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: text/plain; charset=utf-8
Date: Wed, 02 Mar 2022 16:20:39 GMT
Strict-Transport-Security: max-age=31536000

2022/03/02 11:20:39 <-- 401 https://index.docker.io/v2/ekcasey/test1/blobs/uploads/?from=ekcasey%2Ftest&mount=sha256%3A70915a141de0b1673d9319034ab5ea6286f3ce962939d84ac00943ee7ae56730 (17.079894ms)
2022/03/02 11:20:39 HTTP/1.1 401 Unauthorized
Content-Length: 299
Content-Type: application/json
Date: Wed, 02 Mar 2022 16:20:39 GMT
Docker-Distribution-Api-Version: registry/2.0
Strict-Transport-Security: max-age=31536000
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:ekcasey/test1:pull,push repository:ekcasey/test:pull",error="insufficient_scope"

Thanks @dmikusa-pivotal for discovering that using space-separated scopes in the token exchange does work, but using multiple scope parameters ~(as described in the documentation)~ does not work.

Edit: Re-reading this I see that the Www-Authenticate header returned actually has a single scope parameter, so perhaps we should be handling this differently, but the initial request asks for equivalent scopes:

scope=repository%3Aekcasey%2Ftest1%3Apush%2Cpull&scope=repository%3Aekcasey%2Ftest%3Apull&scope=repository%3Aekcasey%2Ftest1%3Apull%2Cpush

But Docker Hub seems to ignore them?

Task List

• [x] This is NOT a security issue
• [x] I do NOT have a Docker subscription
• [x] I have looked through other issues and they do NOT apply to me

[sudo-bmitch]

regclient is also affected by this.

[ekcasey]

This is a pretty big deal for the buildpacks project and platforms built of top of it all of which are suddenly broken for dockerhub users.

[imjasonh]

This is also affecting users of a number of other projects, like Kaniko, cosign, ko, crane, probably imgpkg, likely others.

Please consider rolling back this change until we can coordinate and propagate this new behavior across these projects.

[thaJeztah]

Thanks for reporting; I see the team is working on this; looks like a fix was merged (not sure if it's deployed already)

[vladaionescu]

This is affecting users of Earthly too

WARN: (Load metadata linux/amd64) pull access denied, repository does not exist or may require authorization: authorization status: 401: authorization failed
Error: build target: build main: bkClient.Build: failed to solve: pull access denied, repository does not exist or may require authorization: authorization status: 401: authorization failed

[joeforshaw]

Also believe this is affecting us at Servd too. https://status.servd.host

[jcarter3]

The fix for this should be deployed, please let us know if the issue persists.

[Tochemey]

@jcarter3 the issue still persists

[joeforshaw]

@jcarter3 Yep still seeing the error persisting our end too.

[ramiayoub-priv]

Seems to be affecting Kubernetes image pulls from a private repo too, cannot pull images atm Failed to pull image "privateregistry/redacted>": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/privateregistry/redacted": failed to resolve reference "docker.io/privateregistry/redacted": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

[jonjohnsonjr]

I see some evidence that this is at least partially fixed.

Earlier:

$ curl -s "https://auth.docker.io/token?scope=repository:library/nginx:pull&scope=repository:library/mysql:pull%20repository:library/ubuntu:pull&service=registry.docker.io" | jq -r .access_token |  cut -d . -f2 | base64 -d | jq .access
[
  {
    "type": "repository",
    "name": "library/nginx",
    "actions": [
      "pull"
    ],
    "parameters": {
      "pull_limit": "100",
      "pull_limit_interval": "21600"
    }
  }
]

Now:

 $ curl -s "https://auth.docker.io/token?scope=repository:library/nginx:pull&scope=repository:library/mysql:pull%20repository:library/ubuntu:pull&service=registry.docker.io" | jq -r .access_token |  cut -d . -f2 | base64 -d | jq .access
base64: invalid input
[
  {
    "type": "repository",
    "name": "library/mysql",
    "actions": [
      "pull"
    ],
    "parameters": {
      "pull_limit": "100",
      "pull_limit_interval": "21600"
    }
  },
  {
    "type": "repository",
    "name": "library/nginx",
    "actions": [
      "pull"
    ],
    "parameters": {
      "pull_limit": "100",
      "pull_limit_interval": "21600"
    }
  },
  {
    "type": "repository",
    "name": "library/ubuntu",
    "actions": [
      "pull"
    ],
    "parameters": {
      "pull_limit": "100",
      "pull_limit_interval": "21600"
    }
  }
]

[mattgrayisok]

We're failing to pull all images in k8s too which I assume is related to this:

"docker.io/library/busybox:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

[mdkent]

Seeing the same here on Buildkite:

[2022-03-02T20:28:47Z]  > [1/9] FROM docker.io/library/ubuntu:18.04:
[2022-03-02T20:28:47Z] ------
[2022-03-02T20:28:47Z] pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

[Lp-Francois]

Same here with docker login on mac, and API key:

Error saving credentials: error storing credentials - err: exit status 1, out: Post "http://ipc/registry/credstore-updated": dial unix Library/Containers/com.docker.docker/Data/backend.sock: connect: connection refused

Several K8s clusters failing to pull images (private & public): Users of Digital Ocean managed K8s might be affected

Failed to pull image "redis:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/redis:latest": failed to resolve reference "docker.io/library/redis:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

And bitbucket pipelines using docker.

Hope it can help 🙏

[jgreat]

If somebody from docker is working on it and knows this is an issue, could you please update the https://status.docker.com/ page.

[joeforshaw]

We're starting to see pulls succeed again 🙌

[jcarter3]

Previous changes have all been reverted at this point as we continue to investigate this issue.

[Matroxt]

Thanks for resolving the issue ❤️

To add to what jgreat said, the communication from Docker really needs to improve in case of outages.

Theses things happens, we get that, but I shouldn't need to search for a obscure Github issue 3 hours into the outage to get updates on something that has that big of an impact.

I trust that you'll bring this up internally to improve on this matter.

[jcarter3]

Sorry for the issues this caused. We've identified the root cause - the exact nature of this bug made it difficult to pinpoint and we are experimenting with ways that we can monitor this going forward.

[joeforshaw]

@jcarter3 We're currently seeing this issue reappear. This is an error from one of our kubernetes pods thats failing to pull a private image (image name partially changed):

Failed to pull image "docker.io/servdhosting/xxx": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/servdhosting/xxx": failed to resolve reference "docker.io/servdhosting/xxx": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

It was working earlier today, seems to have started in the past hour or two.

[jcarter3]

@joeforshaw This is likely a different issue as there haven't been any changes made to the services in some time. Can you replicate this with a different user? Is it a new/different tag that is not working?

[joeforshaw]

Sorry @jcarter3, looks like a false alarm. User error! 🤦‍♂️

We made some security upgrades a few weeks ago which prevented our docker hub auth requests from being sent in certain circumstances. Apologies!

[technicallyjosh]

Phew! We were just looking into that 😅. Thanks for the update @joeforshaw

[joeforshaw]

Sorry about that! Customers are the worst. 🤪