Error reading manifest latest in registry... : unauthorized: authentication required

logopk commented 4 years ago

Behaviour

We have had a similar issue like this a while ago #88 and we fixed it with the below config. But apparently it's no longer working. I have downgraded all the way to 4.0.0 but without success.

Steps to reproduce this issue

environment with private registry login
labels on container with private registry image
run diun

Expected behaviour

request to registry is authorised and image status can be verified

Actual behaviour

diun-mbp2_1 | Wed, 02 Sep 2020 16:46:09 CEST ERR Cannot list tags from registry error="Error reading manifest latest in registry.:5000/logopk/cloudflared: unauthorized: authentication required" image=registry.:5000/logopk/cloudflared:latest provider=docker

Configuration

Diun version : 4.5.0
Platform (windows/linux) : linux/boot2docker Mac
System info (type uname -a) : Linux default 4.19.130-boot2docker #1 SMP Mon Jun 29 23:52:55 UTC 2020 x86_64 GNU/Linux

Paste your configuration files here

  diun-mbp2:
    image: crazymax/diun
    hostname: diun-mbp2
    volumes:
      - "./data:/data"
      - ./certs.pem:/etc/ssl/certs/ca-certificates.crt:ro
      - "./file.yml:/file.yml:ro"
      - "/var/run/docker.sock:/var/run/docker.sock"
    environment:
      - "TZ=Europe/Paris"
      - "LOG_LEVEL=info"
      - "LOG_JSON=false"
      - "DIUN_WATCH_WORKERS=20"
      - "DIUN_WATCH_SCHEDULE=*/30 * * * *"
      - "DIUN_PROVIDERS_DOCKER=true"
      - "DIUN_PROVIDERS_DOCKER_WATCHBYDEFAULT=true"
      - "DIUN_PROVIDERS_DOCKER_WATCHSTOPPED=true"
      - "DIUN_PROVIDERS_FILE_FILENAME=/file.yml"
      - "DIUN_REGOPTS_myregistry_NAME=myregistry"
      - "DIUN_REGOPTS_myregistry_USERNAME=logo"
      - "DIUN_REGOPTS_myregistry_PASSWORD=<redacted>"
    restart: always

  cloudflared:
    container_name: cloudflared
    image: registry.<xxx>:5000/logopk/cloudflared:latest
    restart: unless-stopped
    command: "proxy-dns --address 0.0.0.0 --port 5054 --upstream https://1.1.1.1/.well-known/dns-query --upstream https://1.0.0.1/.well-known/dns-query"
    networks:
      internal:
        ipv4_address: 10.1.0.2
    labels:
      - "diun.regopts_id=myregistry"
      - "diun.watch_repo=true"
      - "diun.max_tags=1"

  cadvisor:
    image: registry.<xxx>:5000/logopk/cadvisor:latest
    container_name: cadvisor
    volumes:
      - /etc/machine-id:/etc/machine-id:ro
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /cgroup:/cgroup:ro #doesn't work on MacOS only for Linux
    restart: unless-stopped
    ports:
      - 8080:8080
    expose:
      - 8080
    networks:
      - monitor-net
    labels:
      - "org.label-schema.group=monitoring"
      - "diun.regopts_id=myregistry"
      - "diun.watch_repo=true"
      - "diun.max_tags=1"

Logs

diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST INF Starting Diun version=4.5.0
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG No configuration file found
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST INF Configuration loaded from 18 environment variable(s)
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG {
diun-mbp2_1  |   "db": {
diun-mbp2_1  |     "path": "/data/diun.db"
diun-mbp2_1  |   },
diun-mbp2_1  |   "watch": {
diun-mbp2_1  |     "workers": 20,
diun-mbp2_1  |     "schedule": "*/30 * * * *",
diun-mbp2_1  |     "firstCheckNotif": false
diun-mbp2_1  |   },
diun-mbp2_1  |   "regopts": [
diun-mbp2_1  |     {
diun-mbp2_1  |       "name": "myregistry",
diun-mbp2_1  |       "selector": "name",
diun-mbp2_1  |       "username": "<redacted>",
diun-mbp2_1  |       "password": "<redacted>",
diun-mbp2_1  |       "insecureTLS": false,
diun-mbp2_1  |       "timeout": 10000000000
diun-mbp2_1  |     }
diun-mbp2_1  |   ],
diun-mbp2_1  |   "providers": {
diun-mbp2_1  |     "docker": {
diun-mbp2_1  |       "tlsVerify": true,
diun-mbp2_1  |       "watchByDefault": true,
diun-mbp2_1  |       "watchStopped": true
diun-mbp2_1  |     },
diun-mbp2_1  |     "file": {
diun-mbp2_1  |       "filename": "/file.yml"
diun-mbp2_1  |     }
diun-mbp2_1  |   }
diun-mbp2_1  | }
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG 1 notifier(s) created
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG 103 entries found in manifest bucket
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG Current database version: 2
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST INF Cron triggered
...
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST INF Found 10 image(s) to analyze provider=docker
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image] Loading registries configuration "/etc/containers/registries.conf"
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image] Trying to access "registry.<xxx>:5000/logopk/cadvisor:latest"
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image] Returning credentials from DockerAuthConfig
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image] Using registries.d directory /etc/containers/registries.d for sigstore configuration
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image]  No signature storage configuration found for registry.<xxx>:5000/logopk/cadvisor:latest
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.<xxx>:5000
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image] GET https://registry.<xxx>:5000/v2/
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image] Trying to access "registry.<xxx>:5000/logopk/cadvisor:latest"
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image] Returning credentials from DockerAuthConfig
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image] Using registries.d directory /etc/containers/registries.d for sigstore configuration
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image]  No signature storage configuration found for registry.<xxx>:5000/logopk/cadvisor:latest
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.<xxx>:5000
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image] GET https://registry.<xxx>:5000/v2/
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image] Ping https://registry.<xxx>:5000/v2/ status 401
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image] GET https://registry.<xxx>:5000/v2/logopk/cadvisor/manifests/latest
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image] Ping https://registry.<xxx>:5000/v2/ status 401
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:04 CEST DBG [containers/image] GET https://registry.<xxx>:5000/v2/logopk/cadvisor/manifests/latest
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:06 CEST DBG [containers/image] Content-Type from manifest GET is "application/json; charset=utf-8"
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:06 CEST DBG [containers/image] Accessing "registry.<xxx>:5000/logopk/cadvisor:latest" failed: Error reading manifest latest in registry.<xxx>:5000/logopk/cadvisor: unauthorized: authentication required
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:06 CEST WRN Cannot get remote manifest error="Cannot create image closer: Error reading manifest latest in registry.<xxx>:5000/logopk/cadvisor: unauthorized: authentication required" image=registry.<xxx>:5000/logopk/cadvisor:latest provider=docker
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] Accessing "registry.<xxx>:5000/logopk/cadvisor:latest" failed: Error reading manifest latest in registry.<xxx>:5000/logopk/cadvisor: unauthorized: authentication required
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST ERR Cannot list tags from registry error="Error reading manifest latest in registry.<xxx>:5000/logopk/cadvisor: unauthorized: authentication required" image=registry.<xxx>:5000/logopk/cadvisor:latest provider=docker
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] Trying to access "registry.<xxx>:5000/logopk/cloudflared:latest"
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] Returning credentials from DockerAuthConfig
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] Using registries.d directory /etc/containers/registries.d for sigstore configuration
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image]  No signature storage configuration found for registry.<xxx>:5000/logopk/cloudflared:latest
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.<xxx>:5000
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] GET https://registry.<xxx>:5000/v2/
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] Trying to access "registry.<xxx>:5000/logopk/cloudflared:latest"
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] Returning credentials from DockerAuthConfig
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] Using registries.d directory /etc/containers/registries.d for sigstore configuration
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image]  No signature storage configuration found for registry.<xxx>:5000/logopk/cloudflared:latest
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.<xxx>:5000
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] GET https://registry.<xxx>:5000/v2/
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] Ping https://registry.<xxx>:5000/v2/ status 401
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] GET https://registry.<xxx>:5000/v2/logopk/cloudflared/manifests/latest
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] Ping https://registry.<xxx>:5000/v2/ status 401
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] GET https://registry.<xxx>:5000/v2/logopk/cloudflared/manifests/latest
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] Content-Type from manifest GET is "application/json; charset=utf-8"
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] Accessing "registry.<xxx>:5000/logopk/cloudflared:latest" failed: Error reading manifest latest in registry.<xxx>:5000/logopk/cloudflared: unauthorized: authentication required
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST WRN Cannot get remote manifest error="Cannot create image closer: Error reading manifest latest in registry.<xxx>:5000/logopk/cloudflared: unauthorized: authentication required" image=registry.<xxx>:5000/logopk/cloudflared:latest provider=docker
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] Content-Type from manifest GET is "application/json; charset=utf-8"
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST DBG [containers/image] Accessing "registry.<xxx>:5000/logopk/cloudflared:latest" failed: Error reading manifest latest in registry.<xxx>:5000/logopk/cloudflared: unauthorized: authentication required
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:09 CEST ERR Cannot list tags from registry error="Error reading manifest latest in registry.<xxx>:5000/logopk/cloudflared: unauthorized: authentication required" image=registry.<xxx>:5000/logopk/cloudflared:latest provider=docker
...
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:17 CEST INF Cron initialized with schedule */30 * * * *
diun-mbp2_1  | Wed, 02 Sep 2020 16:46:17 CEST INF Next run in 13 minutes (2020-09-02 17:00:00 +0200 CEST)

crazy-max commented 4 years ago

@logopk

We have had a similar issue like this a while ago #88 and we fixed it with the below config.

Docker labels have changed since v4. Should be "diun.regopt=myregistry".

diun-mbp2_1 | Wed, 02 Sep 2020 16:46:04 CEST DBG No configuration file found

Also, it looks like your configuration file /file.yml is not loaded. You can use that instead:

  diun-mbp2:
    image: crazymax/diun
    hostname: diun-mbp2
    volumes:
      - "./data:/data"
      - ./certs.pem:/etc/ssl/certs/ca-certificates.crt:ro
      - "./file.yml:/diun.yml:ro"
      - "/var/run/docker.sock:/var/run/docker.sock"
    environment:
      - "TZ=Europe/Paris"
      - "LOG_LEVEL=info"
      - "LOG_JSON=false"
      - "DIUN_WATCH_WORKERS=20"
      - "DIUN_WATCH_SCHEDULE=*/30 * * * *"
      - "DIUN_PROVIDERS_DOCKER=true"
      - "DIUN_PROVIDERS_DOCKER_WATCHBYDEFAULT=true"
      - "DIUN_PROVIDERS_DOCKER_WATCHSTOPPED=true"
      - "DIUN_REGOPTS_myregistry_NAME=myregistry"
      - "DIUN_REGOPTS_myregistry_USERNAME=logo"
      - "DIUN_REGOPTS_myregistry_PASSWORD=<redacted>"
    restart: always

See Configuration file doc about config autoload.

logopk commented 4 years ago

Thanks, @crazy-max, the changed label did the trick.

probably missed that migration step...

crazy-max / diun