Images not downloading - "manifest unknown"

[Stitch10925]

Hey,

First of all, thank you for the project. I have been looking for a lightweight docker image cache that was easy to set up. This seems to tick all the boxes for me. However, I seem to have hit a problem that I don't know how to solve.

Error from the Cache log:

time="2024-09-05T21:33:18.612095902Z" level=info msg="Challenge established with upstream : {https docker.io /v2/ %!s(bool=false) %!s(bool=false) } &{{{%!s(int32=0) %!s(uint32=0)} %!s(uint32=0) %!s(uint32=0) {{} %!s(int32=0)} {{} %!s(int32=0)}} map[https://www.docker.com:443/v2/:[]]}" go.version=go1.20.8 http.request.host="localhost:5001" http.request.id=1a9fac7d-e3aa-47d3-a379-dd55570a0433 http.request.method=GET http.request.remoteaddr="127.0.0.1:39816" http.request.uri="/v2/zadam/trilium/manifests/0.62.4" http.request.useragent="docker/25.0.5 go/go1.21.10 git-commit/e63daec8672d77ac0b2b5c262ef525c7cf17fd20 kernel/6.6.47-0-lts os/linux arch/amd64" vars.name="zadam/trilium" vars.reference=0.62.4

time="2024-09-05T21:33:19.034715401Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown tag=0.62.4" err.message="manifest unknown" go.version=go1.20.8 http.request.host="localhost:5001" http.request.id=1a9fac7d-e3aa-47d3-a379-dd55570a0433 http.request.method=GET http.request.remoteaddr="127.0.0.1:39816" http.request.uri="/v2/zadam/trilium/manifests/0.62.4" http.request.useragent="docker/25.0.5 go/go1.21.10 git-commit/e63daec8672d77ac0b2b5c262ef525c7cf17fd20 kernel/6.6.47-0-lts os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=656.990622ms http.response.status=404 http.response.written=96 vars.name="zadam/trilium" vars.reference=0.62.4

127.0.0.1 - - [05/Sep/2024:21:33:18 +0000] "GET /v2/zadam/trilium/manifests/0.62.4 HTTP/1.1" 404 96 "" "docker/25.0.5 go/go1.21.10 git-commit/e63daec8672d77ac0b2b5c262ef525c7cf17fd20 kernel/6.6.47-0-lts os/linux arch/amd64"

From what I understand it managed to connect to docker hub but could not download the image. The image it is trying to download is "zadam/trilium:0.62.4", which exist on dockerhub:

I also registered the Docker Cache as insecure registry:

This is my compose file:

version: '3.8'

services:
  app:
    image: ghcr.io/lhns/docker-registry-cache:d732b51d
    hostname: packages.REDACTED.lan #port 5000
    ports:
    - 5000:5000
    volumes:
    - vol:/var/lib/registry
    environment:
      CONFIG: |
        [
          "docker.io",
          "ghcr.io"
        ]
    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
        condition: any
        delay: 5s
        window: 5s

volumes:
  vol:
    driver: local
    driver_opts:
      type: "nfs"
      o: "addr=REDACTED,rw,noatime,rsize=8192,wsize=8192,tcp,timeo=14,nfsvers=4"
      device: ":/export/DockerImages"

So I'm not quite sure if I missed something.

[lhns]

Hi, can you show the command you use to pull the image?

[Stitch10925]

Docker pulls the image for me (I'm using docker swarm).

The compose file contains the following:

Manually I tried using this:

[lhns]

Oh so the pull in docker swarm works? When you are using the docker pull command do you use that on one of your swarm machines so that 127.0.0.1 is routed by docker swarm to the registry-cache container?

[Stitch10925]

When using the PULL command without the Docker Cache everything works fine, when I add "127.0.0.1:5000" is when I get het "manifest unknown" error.

As you can see in the screenshots, the "manifest unknown" error is logged in the Docker Cache logging. This tells me:

1. That the node is reaching the Docker Cache and is trying to download from it
2. The Docker Cache for some reason can't download the requested image.

I assume the URL used to download the image by Docker Cache might be incorrectly parsed, but that's just a guess.

[lhns]

Ah I see, Can you try using registry-1.docker.io instead of docker.io in the config?

[Stitch10925]

Using "registry-1.docker.io" worked, thanks!

Not sure if this address can be added as fallback address when images are trying to be pulled from Docker? It might also be useful to add this URL to the example in the Readme file for future reference?

What also would help for debugging in the future, would be a health endpoint. As of now I found no way to test if my node could reach 172.0.0.1:5000, but having an endpoint like "127.0.0.1:5000/health" would allow users to test if they can reach the Docker Cache.