registry:2.8.2 forgets it is configured to do token authentication

[rrayst]

Bug: registry 2.8.2 sometimes forgets it is configured to do token authentication.

When I say "it forgets to do token auth", I mean that curl https://myregistry2.example.com/v2/_catalog -v returns

HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Basic realm="Registry Realm"
X-Content-Type-Options: nosniff
Date: Mon, 28 Aug 2023 14:27:18 GMT
Content-Length: 145

instead of

HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer realm="https://auth.example.com/api/docker-registry-token",service="https://myregistry2.example.com",scope="registry:catalog:*"
X-Content-Type-Options: nosniff
Date: Mon, 28 Aug 2023 14:28:01 GMT
Content-Length: 145

from then on. Only a container restart will fix the problem.

---

Configuration Details:

We are running the registry:2.8.2 docker image on a Kubernetes cluster. (Everything is infrastructure as code.)

The container has the following environment variables set:

REGISTRY_AUTH=token
REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm
REGISTRY_AUTH_HTPASSWD_PATH=/config/htpasswd
REGISTRY_AUTH_TOKEN_REALM=https://auth.example.com/api/docker-registry-token
REGISTRY_AUTH_TOKEN_ISSUER=https://auth.example.com/api
REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE=/config/example-key.pem
REGISTRY_AUTH_TOKEN_SERVICE=https://myregistry2.example.com
REGISTRY_HTTP_SECRET=....

and /config/ contains

version: 0.1
log:
  fields:
    service: registry
storage:
    delete:
        enabled: true
    cache:
        blobdescriptor: inmemory
    filesystem:
        rootdirectory: /var/lib/registry
http:
    addr: :5000
    headers:
        X-Content-Type-Options: [nosniff]
    debug:
        addr: ":5001"
        prometheus:
            enabled: true
            path: "/metrics"
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

so there is nothing fancy there.

The container is started using /bin/registry serve /config/config.yml as command.

The file /config/htpasswd also exists (legacy reasons) and containers 2 users. (But the file should not be considered, as REGISTRY_AUTH=token is set.)

After starting the container, it can run for days without problems (doing token auth). But at some point (after weeks to months), it starts returning Www-Authenticate: Basic [...] instead of Www-Authenticate: Bearer [...]. At this point, there seems to be no other solution than restarting the container.

As a programmer, based on the behaviour described above, I would usually bet on a concurrency issue. With my limited Go experience, I would still say that the code accessing the authentication method seems fine, though.

The bug was already present in the registry:2.7.1 images.

[milosgajdos]

You are mixing 2 AUTH providers: token and httpasswd.

The docs state:

You can configure only one authentication provider.

[milosgajdos]

@rrayst mind closing this as this is a misused configuration case, not an issue with registry.