search

cesanta / docker_auth

Authentication server for Docker Registry 2
Apache License 2.0
1.27k stars 305 forks source link

No contact between Registry Container and Authentication Container - nothing in the logs even in verbose mode showing that they are attempting to talk to each other #390

Closed luckylinux closed 5 months ago

luckylinux commented 5 months ago

I am trying to get this to work with podman.

I started off with: https://github.com/cesanta/docker_auth/blob/main/examples/simple.yml

This also gave some ideas on how to enable anonymous pulls: https://stackoverflow.com/questions/38310906/anonymous-pull-on-docker-repo-in-artifactory

My Compose File for a Registry running behind Traefik Proxy:

version: "3.8"

services:
  docker-local-mirror-registry:
    image: registry:latest
#    pull_policy: "missing"
    container_name: docker-local-mirror-registry
    hostname: docker-local-mirror-registry
    volumes:
      - ~/containers/data/docker-local-mirror-registry:/var/lib/registry
    #  - ~/containers/certificates/letsencrypt/MYDOMAIN.TLD:/cert:ro
      - ~/containers/certificates/docker-local-mirror-auth/cert.pem:/cert/auth/cert.pem:ro
      - ~/containers/config/docker-local-mirror-registry/auth/htpasswd:/auth/htpasswd:ro
      - ~/containers/config/docker-local-mirror-registry/registry:/etc/docker/registry:ro
#    expose:
#      - 80
#    ports:
#      - 5000:5000
    networks:
#      - podman
      - traefik
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.docker-local-mirror-registry-router.rule=Host(`docker.MYDOMAIN.TLD`,`docker-local.MYDOMAIN.TLD`,`docker-local-mirror.MYDOMAIN.TLD`,`docker-local-mirror-registry.MYDOMAIN.TLD`,`docker-images.MYDOMAIN.TLD`,`docker-mirror.MYDOMAIN.TLD`)"

      # Enable both Headers & Authentication Middlewares
      #- "traefik.http.routers.docker-local-mirror-registry-router.middlewares=docker-local-mirror-registry-headers,docker-local-mirror-registry-auth"

      # Headers Middleware
      #- "traefik.http.routers.docker-local-mirror-registry-router.middlewares=docker-local-mirror-registry-headers"
      #- "traefik.http.middlewares.docker-local-mirror-registry-headers.headers.customrequestheaders.Connection=Upgrade"

      # Setup Authentication
      #- "traefik.http.routers.docker-local-mirror-registry-router.middlewares=docker-local-mirror-registry-auth"
      #- "traefik.http.middlewares.docker-local-mirror-registry-auth.basicauth.usersfile=~/containers/config/docker-local-mirror-registry/auth/htpasswd"
      #- "traefik.http.middlewares.docker-local-mirror-registry-auth.basicauth.usersfile=/auth/htpasswd"

      # Setup Service
      - "traefik.http.services.docker-local-mirror-registry-service.loadbalancer.server.port=5000"
      - "traefik.docker.network=traefik"
    environment:
      #- "REGISTRY_AUTH=htpasswd"                               # Disabled since we use docker-local-mirror-auth (cesanta/docker_auth)
      #- "REGISTRY_AUTH_HTPASSWD_REALM=Registry-Realm"          # Disabled since we use docker-local-mirror-auth (cesanta/docker_auth) 
      #- "REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd"           # Disabled since we use docker-local-mirror-auth (cesanta/docker_auth)
      #- "REGISTRY_HTTP_TLS_CERTIFICATE=/cert/fullchain.pem"    # Not needed since Traefik handles this
      #- "REGISTRY_HTTP_TLS_KEY=/cert/privkey.pem"              # Not needed since Traefik handles this 
      - "REGISTRY_HTTP_ADDR=0.0.0.0:5000"
      - "REGISTRY_LOG_LEVEL=debug"
      - "REGISTRY_STORAGE_DELETE_ENABLED=false"
      - "REGISTRY_STORAGE_DELETE_AGE=1344"

  docker-local-mirror-auth:
    image: cesanta/docker_auth:latest
    container_name: docker-local-mirror-auth
    hostname: docker-local-mirror-auth
    volumes:
      - ~/containers/log/docker-local-mirror-auth:/logs
      - ~/containers/config/docker-local-mirror-auth:/config:ro
      - ~/containers/certificates/docker-local-mirror-auth:/cert/auth:ro
    restart: "unless-stopped"
    command: --v=2 --alsologtostderr /config/config.yml
    networks:
      - podman
    #expose:
    #  - 5001
    ports:
      - 5001:5001

networks:
  podman:
    external: true
  traefik:
    external: true

My Registry Configuration:

version: 0.1
auth:
  token:
    #realm: "https://docker-local-mirror-auth:5001/auth"
    #realm: "https://127.0.0.1:5001/auth"
    realm: "https://localhost:5001/auth"
    service: "Docker Registry"
    issuer: "docker-local-mirror-auth"
    rootcertbundle: "/cert/auth/cert.pem"
    autoredirect: false

storage:
  filesystem:
    rootdirectory: /var/lib/registry

http:
  addr: 0.0.0.0:5000
  prefix: ""
  secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
#  tls:
#    certificate: /path/to/x509/public
#    key: /path/to/x509/private
#    clientcas:
#      - /path/to/ca.pem
#      - /path/to/another/ca.pem

My Auth Configuration (docker_auth):

server:
  addr: "0.0.0.0:5001"
  certificate: "/cert/auth/cert.pem"
  key: "/cert/auth/key.pem"

token:
  issuer: "docker-local-mirror-auth"  # Must match issuer in the Registry config.
  expiration: 900

users:
  # Password is specified as a BCrypt hash. Use `htpasswd -nB USERNAME` to generate.
  "PodmanServer15":
    password: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  "UbuntuWorkstation02":
    password: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  # Allow anonymous (no "docker login") access.
  "": {}
acl:
 # - match: {account: "admin"}
 #   actions: ["*"]
 #   comment: "Admin has full access to everything."
 # - match: {account: "test"}
 #   actions: ["pull"]
 #   comment: "User \"test\" can pull stuff."
  - match: {account: "/.+/"}
    actions: ["*"]
    comment: "Logged in users can do anything."
  - match: {account: ""}
    actions: ["pull"]
    comment: "Anonymous users can pull anything."
  # Access is denied by default.

I tried both to access the auth server from localhost, 127.0.0.1 or docker-local-mirror-auth. In theory (which is working at least for PostgreSQL communications with another Container on the same Host), docker-local-mirror-auth is the one that should be used.

No matter what I do, it seems that the Registry never even attemps to contact the Authentication Server. There are no logs on attempted Connection between them. Only an Authorisation Error on the Registry Container ....

Authentication Server Verbose Logs:

I0414 20:08:55.964533       1 main.go:248] docker_auth  build 
I0414 20:08:55.993970       1 main.go:61] Config from /config/config.yml (3 users, 2 ACL static entries)
I0414 20:08:55.994165       1 acl.go:109] Created ACL Authorizer with 2 entries
I0414 20:08:55.994370       1 main.go:112] Cert file: /cert/auth/cert.pem
I0414 20:08:55.994387       1 main.go:113] Key file : /cert/auth/key.pem
I0414 20:08:55.996137       1 main.go:175] Serving on 0.0.0.0:5001

Registry Server Debug Logs:

time="2024-04-14T20:09:01.542983354Z" level=info msg="redis not configured" go.version=go1.20.8 instance.id=3aab758c-5087-4804-98a3-1160a7f1a40d version=2.8.3 
time="2024-04-14T20:09:01.543197954Z" level=info msg="Starting upload purge in 30m0s" go.version=go1.20.8 instance.id=3aab758c-5087-4804-98a3-1160a7f1a40d version=2.8.3 
time="2024-04-14T20:09:01.543426318Z" level=debug msg="configured "token" access controller" go.version=go1.20.8 instance.id=3aab758c-5087-4804-98a3-1160a7f1a40d version=2.8.3 
time="2024-04-14T20:09:01.54437945Z" level=info msg="listening on [::]:5000" go.version=go1.20.8 instance.id=3aab758c-5087-4804-98a3-1160a7f1a40d version=2.8.3 
time="2024-04-14T20:09:13.236681962Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=1f65ffed-bd62-4670-87d4-e14a42733209 http.request.method=GET http.request.remoteaddr=10.89.0.65 http.request.uri="/v2/_catalog" http.request.useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" 
time="2024-04-14T20:09:13.236856477Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=1f65ffed-bd62-4670-87d4-e14a42733209 http.request.method=GET http.request.remoteaddr=10.89.0.65 http.request.uri="/v2/_catalog" http.request.useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" 
10.89.0.65 - - [14/Apr/2024:20:09:13 +0000] "GET /v2/_catalog HTTP/1.1" 401 145 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
time="2024-04-14T20:09:13.802871422Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=85bcb271-c36a-4a1e-9b7e-caf483a03361 http.request.method=GET http.request.remoteaddr=10.89.0.65 http.request.uri="/v2/_catalog" http.request.useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" 
10.89.0.65 - - [14/Apr/2024:20:09:13 +0000] "GET /v2/_catalog HTTP/1.1" 401 145 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
time="2024-04-14T20:09:13.802941288Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=85bcb271-c36a-4a1e-9b7e-caf483a03361 http.request.method=GET http.request.remoteaddr=10.89.0.65 http.request.uri="/v2/_catalog" http.request.useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" 
time="2024-04-14T20:09:13.961217835Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=8aece0c7-16c7-44c5-bc78-143ec803a11a http.request.method=GET http.request.remoteaddr=10.89.0.65 http.request.uri="/v2/_catalog" http.request.useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" 
10.89.0.65 - - [14/Apr/2024:20:09:13 +0000] "GET /v2/_catalog HTTP/1.1" 401 145 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
time="2024-04-14T20:09:13.961281508Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=8aece0c7-16c7-44c5-bc78-143ec803a11a http.request.method=GET http.request.remoteaddr=10.89.0.65 http.request.uri="/v2/_catalog" http.request.useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" 
time="2024-04-14T20:09:14.113753142Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=0f704425-bf0d-4bbd-bfa9-2dfe05740c81 http.request.method=GET http.request.remoteaddr=10.89.0.65 http.request.uri="/v2/_catalog" http.request.useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" 
time="2024-04-14T20:09:14.1138487Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=0f704425-bf0d-4bbd-bfa9-2dfe05740c81 http.request.method=GET http.request.remoteaddr=10.89.0.65 http.request.uri="/v2/_catalog" http.request.useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" 
10.89.0.65 - - [14/Apr/2024:20:09:14 +0000] "GET /v2/_catalog HTTP/1.1" 401 145 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"
time="2024-04-14T20:09:30.303915674Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=8950c1fa-a3b0-405a-a2b7-3b73cc71af7a http.request.method=GET http.request.remoteaddr=10.89.0.65 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.65 - - [14/Apr/2024:20:09:30 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:09:30.303998151Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=8950c1fa-a3b0-405a-a2b7-3b73cc71af7a http.request.method=GET http.request.remoteaddr=10.89.0.65 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:09:31.313517714Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=b2906c30-8ff4-45fb-a569-540a02757d07 http.request.method=GET http.request.remoteaddr=10.89.0.65 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.65 - - [14/Apr/2024:20:09:31 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:09:31.313624748Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=b2906c30-8ff4-45fb-a569-540a02757d07 http.request.method=GET http.request.remoteaddr=10.89.0.65 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:09:32.322157804Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=66dc1eeb-3ed6-416a-be6d-1c7837959e06 http.request.method=GET http.request.remoteaddr=10.89.0.65 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:09:32.322234401Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=66dc1eeb-3ed6-416a-be6d-1c7837959e06 http.request.method=GET http.request.remoteaddr=10.89.0.65 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.65 - - [14/Apr/2024:20:09:32 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:09:33.330632235Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=04e0d79c-0602-4508-9e2a-c084dee4bbef http.request.method=GET http.request.remoteaddr=10.89.0.65 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:09:33.330709829Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=04e0d79c-0602-4508-9e2a-c084dee4bbef http.request.method=GET http.request.remoteaddr=10.89.0.65 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.65 - - [14/Apr/2024:20:09:33 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:10:03.555070624Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=1219aded-9f71-4556-896d-398bee43acfc http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:10:03.555166236Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=1219aded-9f71-4556-896d-398bee43acfc http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:10:03 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:10:04.564897688Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=ae62c4c3-25bd-4263-9de5-b3059eb0d64f http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:10:04 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:10:04.564972961Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=ae62c4c3-25bd-4263-9de5-b3059eb0d64f http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:10:05.574025269Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=673dae2a-172c-41ca-a71f-5b8947f09803 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:10:05.574098818Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=673dae2a-172c-41ca-a71f-5b8947f09803 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:10:05 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:10:06.582016561Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=0c3c6b18-374b-4ba9-a52c-d37a4bd1182e http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:10:06.582074986Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=0c3c6b18-374b-4ba9-a52c-d37a4bd1182e http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:10:06 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:10:36.804758695Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=7e5c5c9e-c333-4fec-9a91-807e77f1b036 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:10:36.804810732Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=7e5c5c9e-c333-4fec-9a91-807e77f1b036 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:10:36 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:10:37.814098716Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=799cd3d8-7460-4f4b-b812-c0ce543b7759 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:10:37 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:10:37.814184574Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=799cd3d8-7460-4f4b-b812-c0ce543b7759 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:10:38.822814982Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=2f937df0-35fc-4aa6-95be-826e9d6ab933 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:10:38 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:10:38.822881064Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=2f937df0-35fc-4aa6-95be-826e9d6ab933 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:10:39.831809697Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=718fa6a1-65e1-4ee5-83f0-7ad7fd91d72e http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:10:39 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:10:39.831884885Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=718fa6a1-65e1-4ee5-83f0-7ad7fd91d72e http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:11:10.056282182Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=65b804a4-89c8-4f24-8922-bca2d1ce98fe http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:11:10 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:11:10.056351311Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=65b804a4-89c8-4f24-8922-bca2d1ce98fe http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:11:11.064569116Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=b66de922-dbe8-46f4-84e4-7a7e06560a28 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:11:11.064638473Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=b66de922-dbe8-46f4-84e4-7a7e06560a28 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:11:11 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:11:12.073444301Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=04e5731f-47a0-42ab-bd06-13ffa1b27366 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:11:12 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:11:12.073507451Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=04e5731f-47a0-42ab-bd06-13ffa1b27366 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:11:13.081924728Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=417f45bf-68ed-4cff-90ec-5cefbe891253 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:11:13.081975095Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=417f45bf-68ed-4cff-90ec-5cefbe891253 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:11:13 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:11:43.304057398Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=affde04f-6c5f-4995-9662-0b01b6a81819 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:11:43 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:11:43.304117038Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=affde04f-6c5f-4995-9662-0b01b6a81819 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:11:44.312031823Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=faeedcdb-be00-4f1b-a32a-5e4985c07fc2 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:11:44 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:11:44.312074407Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=faeedcdb-be00-4f1b-a32a-5e4985c07fc2 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:11:45 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:11:45.326087898Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=511f0c98-7a40-44fd-9369-d45251d24e27 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:11:45.32617021Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=511f0c98-7a40-44fd-9369-d45251d24e27 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:11:46.335367026Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=b37fc75c-b10d-43b1-828c-c108b31ffde5 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:11:46.335419638Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=b37fc75c-b10d-43b1-828c-c108b31ffde5 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:11:46 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:12:16.557166719Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=c297dc3b-d555-4e08-9433-4fae312b7577 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:12:16.557247549Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=c297dc3b-d555-4e08-9433-4fae312b7577 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:12:16 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:12:17.565440616Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=453398ca-0ab1-4e9e-afdb-8eb1c4214fc8 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:12:17 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:12:17.565503711Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=453398ca-0ab1-4e9e-afdb-8eb1c4214fc8 http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:12:18.574213785Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=cb762bb5-4e8a-4f8f-b460-01a5a6b8580d http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:12:18 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:12:18.574289983Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=cb762bb5-4e8a-4f8f-b460-01a5a6b8580d http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
time="2024-04-14T20:12:19.583032994Z" level=debug msg="authorizing request" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=a3435b81-8da2-4d8f-a62a-c71aba5fcfcf http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)" 
10.89.0.66 - - [14/Apr/2024:20:12:19 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "containers/5.29.2 (github.com/containers/image)"
time="2024-04-14T20:12:19.583141594Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.20.8 http.request.host=docker.MYDOMAIN.TLD http.request.id=a3435b81-8da2-4d8f-a62a-c71aba5fcfcf http.request.method=GET http.request.remoteaddr=10.89.0.66 http.request.uri="/v2/" http.request.useragent="containers/5.29.2 (github.com/containers/image)"

Surely I'm missing something very simple in the config ...

It does NOT seem like a Network Error at least. The Registry doesn't even try to contact the Authentication server. There is no trace of that.

tersmitten commented 5 months ago

As far as I know there is no communication between two containers. The registry just adds realm to the Www-Authenticate header. The client (curl, docker login) is just using that to authenticate. 

# curl --head -sSL http://127.0.1.1:5000/v2/;
HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer realm="http://127.0.1.1:5001/auth",service="Docker Registry"
Date: Thu, 25 Apr 2024 11:43:56 GMT
Content-Length: 87
luckylinux commented 5 months ago

As far as I know there is no communication between two containers. The registry just adds realm to the Www-Authenticate header. The client (curl, docker login) is just using that to authenticate.

# curl --head -sSL http://127.0.1.1:5000/v2/;
HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Bearer realm="http://127.0.1.1:5001/auth",service="Docker Registry"
Date: Thu, 25 Apr 2024 11:43:56 GMT
Content-Length: 87

Weird ... Nothing seemed to be happening in my case. Right now I had to fall back to NO AUTH at all because it just didn't work.

But maybe I'm pointing at the wrong "registry" when I try pulls ? Should I point (e.g. docker pull) to the Registry itself or the Authentication Container ? I thought it should be the Registry itself (as usual), but maybe I got that wrong.

tersmitten commented 5 months ago

I would start with curl. And if that seems alright use can do a docker login 127.0.1.1:5000. And if that works a docker pull 127.0.1.1:5000/foo/bar.

luckylinux commented 5 months ago

So I made some progress. One issue is that I had realm: "https://127.0.0.1:5001/auth"

Whereas I'm using Traefik Proxy, so I should have been using just the Internal Hostname AND without TLS/HTTPS: realm: "http://docker-local-mirror-auth:5001/auth"

However, it still fails in the end. It seems that the HOST System wants to Contact docker-local-mirror-auth directly, when issueing podman login docker.MYDOMAIN.TLD.

I thought the communication would have been like this: HOST -> Docker Local Registry [ --> INTERNAL COMMUNICATION --> ] Docker Auth

But maybe I need to expose also the Docker Auth Service over TLS (via Traefik) on a publicly accessible Hostname (just I did for the "main" Registry at docker.MYDOMAIN.TLD) ?

techknowlogick commented 5 months ago

Yup, that's correct. This doesn't communicate with the registry at all. The client will be directed from the registry to the with server, then with server will do the auth process and return a token, and finally the client will use that token and provide it to the registry. As for the realm, it needs to be accessible by the client as that is what the client is told to fetch the token from

luckylinux commented 5 months ago

And if I setup another Traefik Configuration for the new subdomain I createed, I get this:

Error: authenticating creds for "docker.MYDOMAIN.TLD": Requesting bearer token: invalid status code from registry 504 (Gateway Timeout)
luckylinux commented 5 months ago

Whereas now I am getting this, but I don't think I changed anything:

Error: authenticating creds for "docker.MYDOMAIN.TLD": Requesting bearer token: invalid status code from registry 400 (Bad Request)

Again absolutely nothing in the Auth Container Logs besides the normal Startup.

As for the Docker Registry Container, it's filled with completely useless logs.

Is it a case of the Proxy needing to setup some X-Forwarded-to headers etc ?

Or having to enable autoredirect in the Registry Configuration ?

EDIT 1: this could be related to the fact that I'm trying to do podman login from the HOST where the Containers are running.

Traefik logs show a bit more self-explanatory message, since the Traefik TLS Certificate is valid for *.MYDOMAIN.TLD, not localhost:

time="2024-05-05T19:59:03Z" level=debug msg="Serving default certificate for request: \"localhost\""
time="2024-05-05T19:59:03Z" level=debug msg="http: TLS handshake error from 10.89.0.38:54112: remote error: tls: bad certificate"

EDIT 2: actually I am getting that TLS warning with all Connections, weird. I can access the Traefik Dashboard without Issues though (valid LetsEncrypt Certificate) ...

luckylinux commented 5 months ago

I could successfully login now (on the HOST where the Containers are running) by DISABLING TLS in docker_auth Container according to https://github.com/cesanta/docker_auth/blob/main/examples/non_tls.yml

(basically moving Certificates & Key from the server section to the token Section)

In Docker Registry Container Configuration I still have HTTPS going through Traefik: realm: "https://docker-auth.MYDOMAIN.TLD/auth"

And in Compose File I had added the following in the middlewares section of registry (unsure if this had any effect at all though). Full compose.yml file for reference:

version: "3.8"

services:
  docker-local-mirror-registry:
    image: registry:latest
    pull_policy: "missing"
    container_name: docker-local-mirror-registry
    restart: "unless-stopped"
    volumes:
      - ~/containers/data/docker-local-mirror-registry:/var/lib/registry
      - ~/containers/certificates/docker-local-mirror-auth/cert.pem:/cert/auth/cert.pem:ro
      - ~/containers/config/docker-local-mirror-registry/registry:/etc/docker/registry:ro
    networks:
      - traefik
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.docker-local-mirror-registry-router.rule=Host(`docker.MYDOMAIN.TLD`)"

      # Headers Middleware
      - "traefik.http.routers.docker-local-mirror-registry-router.middlewares=docker-local-mirror-registry-headers,docker-local-mirror-registry-forwardauth"
      - "traefik.http.middlewares.docker-local-mirror-registry-headers.headers.customrequestheaders.Connection=Upgrade"

      - "traefik.http.middlewares.docker-local-mirror-registry-forwardauth.forwardauth.address=https://docker-auth.MYDOMAIN.TLD/auth"
      - "traefik.http.middlewares.docker-local-mirror-registry-forwardauth.forwardauth.trustforwardheader=true"
      - "traefik.http.middlewares.docker-local-mirror-registry-forwardauth.forwardauth.authresponseheaders=X-Forwarded-User"

      # Setup Service
      - "traefik.http.services.docker-local-mirror-registry-service.loadbalancer.server.port=5000"
      - "traefik.docker.network=traefik"
    environment:
      - "REGISTRY_HTTP_ADDR=0.0.0.0:5000"
      - "REGISTRY_LOG_LEVEL=debug"
      - "REGISTRY_STORAGE_DELETE_ENABLED=false"
      - "REGISTRY_STORAGE_DELETE_AGE=1344"
      - "REGISTRY_HTTP_SECRET=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

  docker-local-mirror-auth:
    image: cesanta/docker_auth:latest
    pull_policy: "missing"
    container_name: docker-local-mirror-auth
    volumes:
      - ~/containers/log/docker-local-mirror-auth:/logs
      - ~/containers/config/docker-local-mirror-auth:/config:ro
      - ~/containers/certificates/docker-local-mirror-auth:/cert/auth:ro
    restart: "unless-stopped"
    command: --v=2 --alsologtostderr /config/config.yml
    networks:
      - traefik
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.docker-local-mirror-auth-router.rule=Host(`docker-auth.MYDOMAIN.TLD`)"

      # Headers Middleware
      - "traefik.http.routers.docker-local-mirror-auth-router.middlewares=docker-local-mirror-auth-headers"
      - "traefik.http.middlewares.docker-local-mirror-auth-headers.headers.customrequestheaders.Connection=Upgrade"

      # Setup Service
      - "traefik.http.services.docker-local-mirror-auth-service.loadbalancer.server.port=5001"

networks:
  traefik:
    external: true

And finally also the logs of docker-local-mirror-auth show that indeed some Authentication is taking place.

Finally it seems to be working now :+1:.

EDIT 1: Unauthenticated Pulls seems to be working correctly, but I cannot view the Catalog via the web Browser (JSON) anymore. Is this normal ?

EDIT 2: Fixed link (the correct one to use BEHIND Traefix Proxy is https://github.com/cesanta/docker_auth/blob/main/examples/non_tls.yml)

techknowlogick commented 5 months ago

Glad to hear. I'll close this now :) if you have any other problems, feel free to open a new issue.

luckylinux commented 5 months ago

@techknowlogick Maybe consider making a "Traefik Example" (or for other Proxies: Caddy etc) available as well. To be honest it was a major Headache for me to get it working (mostly to not correctly understand the principle that BOTH the Registry and the Auth system needs to be accessible externally by the Client that wants to authenticate).

Once the Host/Domain external accessibility was fixed, I was getting an Error: "Client sent an HTTP request to an HTTPS server."

Found this on StackOverflow: https://stackoverflow.com/questions/77890357/i-got-the-error-client-sent-an-http-request-to-an-https-server-even-the-url-i

After that, I turned OFF TLS as explained above for the Auth Container (TLS is provided by Traefik anyways), then it started working :+1:.

I know there are like 300 combinations possible between Auth backend and Proxy etc, but a folder "traefik" for instance with the Compose file & the non-tls example would go a long way in my View ...