Problem pulling docker images from repo

[pvannierop]

Hi there, thanks for making k3d-registry-dockerd available!

When deploying the registry locally as specified in the README.md, it appears that the repo is used by k3d, but that it is not able to pull the image from Dockerhub. Perhaps it has no access the docker API mounted in the container?

2024/07/19 10:23:43 500 Internal Server Error - HEAD /v2/cbioportal/cbioportal/manifests/latest?ns=docker.io - (3.530113ms)
2024/07/19 10:23:44 404 Not Found - GET /v2/cbioportal/cbioportal/manifests/sha256:1b112823d4153ab65777509fe829c77d805c300e9c99cc4244f9060b6683a8a9?ns=docker.io - (35.437µs)
2024/07/19 10:23:45 404 Not Found - GET /v2/cbioportal/cbioportal/manifests/sha256:be5dfa6563eb5c09b3e915dab98fe9e59196a5941e22b46337cb6f66af84176d?ns=docker.io - (33.203µs)
2024/07/19 10:23:45 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:e946969db3439d19f03aa6bb01020196498bf0342874cace769c049f34a8992b?ns=docker.io - (20.713µs)
2024/07/19 10:23:46 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:8ec0d02fe6614e2b9257bb6c98ce0600f90c5456908c97de806e9c957321a8f1?ns=docker.io - (18.472µs)
2024/07/19 10:23:46 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:9b857f539cb142c9aa2201a17bb8e1cd5cf12edd4a65adf5732fe9f4343964cf?ns=docker.io - (15.665µs)
2024/07/19 10:23:46 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:0ff50609e3ed14c7c3740b327b78aacfe1ea1ee52420196ee57d2b4319ae0936?ns=docker.io - (40.627µs)
2024/07/19 10:23:50 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:b0d053b8dd8bc02c6ecc527c86cf11615cdddfe5c05058f9aeceb20ed29596fd?ns=docker.io - (46.923µs)
2024/07/19 10:23:50 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:b05b6f2f82698bbf2d7ba9eef538debf8e4566b9fcaa945aa32d717b5ad687ab?ns=docker.io - (49.228µs)
2024/07/19 10:23:50 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:3db787cb70aa5c0ac8286e2f3e140189adad0cfe77f350fdf8d27d82d1f00a5f?ns=docker.io - (185.377µs)
2024/07/19 10:23:51 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:02e4bfb717b9562af6f193928383702e3498a00d2f67d91ae48c570a6e139261?ns=docker.io - (41.442µs)
2024/07/19 10:23:51 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:824caa0126073860cdd7c14d30a810a97cb26a7f1260667eea4145547ff02068?ns=docker.io - (44.471µs)
2024/07/19 10:24:12 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:9b84b2fa5f99d66a91427ccab39a50b20f6727bc5c3993c9ebc3416fc4d3ddcc?ns=docker.io - (63.553µs)
2024/07/19 10:24:13 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:da713b7a4bb08e2e409c8313d9f9f21bc442bf8889cc73a07f0a4012e50bc4a6?ns=docker.io - (160.917µs)
2024/07/19 10:24:14 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:f3d193b9ec3638f83d6fda73326ac2d8e7038c6be650b8e4e930b451209ba496?ns=docker.io - (185.896µs)
2024/07/19 10:24:14 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:c9066f4ddafee50d9e417f7abf8e20656d573992b4976dce906b1f820fc6401e?ns=docker.io - (105.22µs)
2024/07/19 10:24:14 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:f3f46f40ae5fc894ee9a8da377a7c69e9cbc908d8ec79b9adbecb7063447c73c?ns=docker.io - (100.872µs)
2024/07/19 10:24:14 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:ef0077df3483bfe0495a7eb074cce562a92d8fce88fd599442fa6d429ec601b1?ns=docker.io - (218.671µs)
2024/07/19 10:24:15 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:7cd012b95d30e66212554cc7999530aa7def7a5ce544d33def0c3258f4c0e99c?ns=docker.io - (79.12µs)
2024/07/19 10:24:19 404 Not Found - GET /v2/cbioportal/cbioportal/blobs/sha256:f072df4ee3ae4b0001ec309a06ca1b6e4d213d26184fc2b3abf535e3d105dbcd?ns=docker.io - (59.542µs)

Would you know how to remedy this problem?

[pvannierop]

@ligfx I have observed the same problem on my local Windows system (WSL2) as wel as on our CI server (Linux).

[ligfx]

Hi @pvannierop, can you share your full config for k3d?

[ligfx]

You can also try running against a new tag I just pushed, which has improved error logging and should be able to tell you what's going wrong: docker pull ligfx/k3d-registry-dockerd:dev

[pvannierop]

I am now using ligfx/k3d-registry-dockerd:dev.

When the registry container has started, it shows the following log:

2024/07/25 03:14:15 Listening on :5000
2024/07/25 03:14:30 500 Internal Server Error - HEAD /v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io - (14.235062ms) - json: cannot unmarshal object into Go value of type []main.DockerImageInfo
2024/07/25 03:14:30 500 Internal Server Error - HEAD /v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io - (1.836967ms) - json: cannot unmarshal object into Go value of type []main.DockerImageInfo
2024/07/25 03:14:30 500 Internal Server Error - HEAD /v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io - (10.493243ms) - json: cannot unmarshal object into Go value of type []main.DockerImageInfo
2024/07/25 03:14:31 404 Not Found - GET /v2/rancher/mirrored-pause/manifests/sha256:74c4244427b7312c5b901fe0f67cbc53683d06f4f24c6faee65d4182bf0fa893?ns=docker.io - (81.198µs)
2024/07/25 03:14:32 404 Not Found - GET /v2/rancher/mirrored-pause/manifests/sha256:c2280d2f5f56cf9c9a01bb64b2db4651e35efd6d62a54dcfc12049fe6449c5e4?ns=docker.io - (45.87µs)
2024/07/25 03:14:32 404 Not Found - GET /v2/rancher/mirrored-pause/blobs/sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee?ns=docker.io - (59.541µs)
2024/07/25 03:14:32 404 Not Found - GET /v2/rancher/mirrored-pause/blobs/sha256:fbe1a72f5dcd08ba4ca3ce3468c742786c1f6578c1f6bb401be1c4620d6ff705?ns=docker.io - (32.556µs)
2024/07/25 03:14:33 500 Internal Server Error - HEAD /v2/rancher/mirrored-metrics-server/manifests/v0.6.3?ns=docker.io - (6.723003ms) - json: cannot unmarshal object into Go value of type []main.DockerImageInfo
2024/07/25 03:14:33 500 Internal Server Error - HEAD /v2/rancher/mirrored-coredns-coredns/manifests/1.10.1?ns=docker.io - (7.703354ms) - json: cannot unmarshal object into Go value of type []main.DockerImageInfo
2024/07/25 03:14:33 500 Internal Server Error - HEAD /v2/rancher/local-path-provisioner/manifests/v0.0.24?ns=docker.io - (5.835222ms) - json: cannot unmarshal object into Go value of type []main.DockerImageInfo
2024/07/25 03:14:34 404 Not Found - GET /v2/rancher/mirrored-metrics-server/manifests/sha256:c2dfd72bafd6406ed306d9fbd07f55c496b004293d13d3de88a4567eacc36558?ns=docker.io - (47.879µs)
2024/07/25 03:14:34 404 Not Found - GET /v2/rancher/local-path-provisioner/manifests/sha256:5bb33992a4ec3034c28b5e0b3c4c2ac35d3613b25b79455eb4b1a95adc82cdc0?ns=docker.io - (66.48µs)
2024/07/25 03:14:34 404 Not Found - GET /v2/rancher/mirrored-coredns-coredns/manifests/sha256:a11fafae1f8037cbbd66c5afa40ba2423936b72b4fd50a7034a7e8b955163594?ns=docker.io - (42.632µs)
2024/07/25 03:14:35 404 Not Found - GET /v2/rancher/local-path-provisioner/manifests/sha256:b7dea5221f06f6feed7788db0ad6b024a433c8f55533bd6cc792dc2079ff9ad2?ns=docker.io - (52.246µs)
2024/07/25 03:14:35 404 Not Found - GET /v2/rancher/mirrored-metrics-server/manifests/sha256:3e7fce61907e003ef8734cc6a39195228002d9b2d64c3fe3b7593f9d448ea133?ns=docker.io - (50.828µs)
2024/07/25 03:14:35 404 Not Found - GET /v2/rancher/mirrored-coredns-coredns/manifests/sha256:be7652ce0b43b1339f3d14d9b14af9f588578011092c1f7893bd55432d83a378?ns=docker.io - (36.824µs)
2024/07/25 03:14:35 404 Not Found - GET /v2/rancher/local-path-provisioner/blobs/sha256:b29384aeb4b13e047448ccfd312c52b4d023abcbbaafcab174293a97821dddb0?ns=docker.io - (32.95µs)
2024/07/25 03:14:35 404 Not Found - GET /v2/rancher/mirrored-metrics-server/blobs/sha256:817bbe3f2e5179b5fa7dd4085c87aab1faaf9dc879e3b1d05796901a006f58ab?ns=docker.io - (29.765µs)
2024/07/25 03:14:35 404 Not Found - GET /v2/rancher/mirrored-coredns-coredns/blobs/sha256:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc?ns=docker.io - (48.177µs)
2024/07/25 03:14:35 404 Not Found - GET /v2/rancher/mirrored-metrics-server/blobs/sha256:b438aade392267e4f99f3bac4b6c46de104f45b105c930014a9045b426b1cec1?ns=docker.io - (32.54µs)
2024/07/25 03:14:35 404 Not Found - GET /v2/rancher/mirrored-metrics-server/blobs/sha256:10f855b03c8aee4fb0b9b7031c333640d684bd9ee6045f11f9892c7fea394701?ns=docker.io - (57.46µs)
2024/07/25 03:14:35 404 Not Found - GET /v2/rancher/mirrored-metrics-server/blobs/sha256:fe5ca62666f04366c8e7f605aa82997d71320183e99962fa76b3209fdfbb8b58?ns=docker.io - (31.564µs)
2024/07/25 03:14:35 404 Not Found - GET /v2/rancher/local-path-provisioner/blobs/sha256:43dba5c51b824080d3cca8d56303454d8ef63adb87a0b2efc428df591a4e448c?ns=docker.io - (31.633µs)
2024/07/25 03:14:35 404 Not Found - GET /v2/rancher/local-path-provisioner/blobs/sha256:7c94c374831ec9e1a47fb11de85ddb6c407da017d3f8b7a308f762bec16e3ff0?ns=docker.io - (25.155µs)
2024/07/25 03:14:35 404 Not Found - GET /v2/rancher/local-path-provisioner/blobs/sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c?ns=docker.io - (28.291µs)
2024/07/25 03:14:35 404 Not Found - GET /v2/rancher/mirrored-coredns-coredns/blobs/sha256:3799eae1a077913c39123d0f4fe4e16243237e7ee94cd84874ea755ec930805a?ns=docker.io - (34.195µs)
2024/07/25 03:14:35 404 Not Found - GET /v2/rancher/mirrored-coredns-coredns/blobs/sha256:25b7032c281a433b92d09930f3a03c0f7382c27eb69ae7f35addf2e3853dbba7?ns=docker.io - (27.845µs)
2024/07/25 03:14:36 404 Not Found - GET /v2/rancher/mirrored-metrics-server/blobs/sha256:fcb6f6d2c9986d9cd6a2ea3cc2936e5fc613e09f1af9042329011e43057f3265?ns=docker.io - (36.573µs)
2024/07/25 03:14:36 404 Not Found - GET /v2/rancher/local-path-provisioner/blobs/sha256:08094466324e3e2f26a4a8d198e89a7e8186ffe5765acbc671a591aa1048b688?ns=docker.io - (34.956µs)
2024/07/25 03:14:36 404 Not Found - GET /v2/rancher/mirrored-metrics-server/blobs/sha256:e8c73c638ae9ec5ad70c49df7e484040d889cca6b4a9af056579c3d058ea93f0?ns=docker.io - (36.738µs)
2024/07/25 03:14:36 404 Not Found - GET /v2/rancher/mirrored-metrics-server/blobs/sha256:1e3d9b7d145208fa8fa3ee1c9612d0adaac7255f1bbc9ddea7e461e0b317805c?ns=docker.io - (34.448µs)
2024/07/25 03:14:36 404 Not Found - GET /v2/rancher/mirrored-metrics-server/blobs/sha256:4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f?ns=docker.io - (62.853µs)
2024/07/25 03:14:36 404 Not Found - GET /v2/rancher/mirrored-metrics-server/blobs/sha256:7c881f9ab25e0d86562a123b5fb56aebf8aa0ddd7d48ef602faf8d1e7cf43d8c?ns=docker.io - (36.863µs)
2024/07/25 03:14:36 404 Not Found - GET /v2/rancher/mirrored-metrics-server/blobs/sha256:5627a970d25e752d971a501ec7e35d0d6fdcd4a3ce9e958715a686853024794a?ns=docker.io - (34.245µs)
2024/07/25 03:14:36 404 Not Found - GET /v2/rancher/mirrored-metrics-server/blobs/sha256:d7f870112e68a947c3e58ef5d7fe1e246c70cb07e7ffd598605e563f3c796436?ns=docker.io - (59.447µs)

This is my k3d config:

# See:
# - https://github.com/k3d-io/k3d/issues/19#issuecomment-1967513596
# - https://github.com/ligfx/k3d-registry-dockerd
apiVersion: k3d.io/v1alpha5
kind: Simple
registries:
  create:
    image: ligfx/k3d-registry-dockerd:dev
    proxy:
      remoteURL: "*"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
options:
  k3s:
    extraArgs:
      - arg: --disable=cloud-controller
        nodeFilters:
          - server:*
      - arg: --disable=helm-controller
        nodeFilters:
          - server:*
      - arg: --disable=traefik
        nodeFilters:
          - server:*
      - arg: --disable=servicelb
        nodeFilters:
          - server:*

I start my cluster with:

 k3d cluster create test --config .github/ci_config/k3d-local-image-repo.yaml 

[ligfx]

@pvannierop looks like it's getting an unexpected response from Docker 🤔 I pushed a new :dev tag that includes code to log out the full JSON response when it hits an error so we can see what's happening — would you mind trying again?

[pvannierop]

Yes, off course. I will test it and report back.

[pvannierop]

@ligfx I got this:

2024/07/26 10:29:45 Listening on :5000
2024/07/26 10:29:59 500 Internal Server Error - HEAD /v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io - (5.480515ms) - json: cannot unmarshal object into Go value of type []main.DockerImageInfo while parsing: {"message":"client version 1.45 is too new. Maximum supported API version is 1.44"}

2024/07/26 10:29:59 500 Internal Server Error - HEAD /v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io - (16.908158ms) - json: cannot unmarshal object into Go value of type []main.DockerImageInfo while parsing: {"message":"client version 1.45 is too new. Maximum supported API version is 1.44"}

2024/07/26 10:29:59 500 Internal Server Error - HEAD /v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io - (13.805839ms) - json: cannot unmarshal object into Go value of type []main.DockerImageInfo while parsing: {"message":"client version 1.45 is too new. Maximum supported API version is 1.44"}

2024/07/26 10:30:00 404 Not Found - GET /v2/rancher/mirrored-pause/manifests/sha256:74c4244427b7312c5b901fe0f67cbc53683d06f4f24c6faee65d4182bf0fa893?ns=docker.io - (58.533µs)
2024/07/26 10:30:01 404 Not Found - GET /v2/rancher/mirrored-pause/manifests/sha256:c2280d2f5f56cf9c9a01bb64b2db4651e35efd6d62a54dcfc12049fe6449c5e4?ns=docker.io - (37.638µs)
2024/07/26 10:30:01 404 Not Found - GET /v2/rancher/mirrored-pause/blobs/sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee?ns=docker.io - (32.723µs)
2024/07/26 10:30:01 404 Not Found - GET /v2/rancher/mirrored-pause/blobs/sha256:fbe1a72f5dcd08ba4ca3ce3468c742786c1f6578c1f6bb401be1c4620d6ff705?ns=docker.io - (28.276µs)
2024/07/26 10:30:01 500 Internal Server Error - HEAD /v2/rancher/mirrored-metrics-server/manifests/v0.6.3?ns=docker.io - (7.928604ms) - json: cannot unmarshal object into Go value of type []main.DockerImageInfo while parsing: {"message":"client version 1.45 is too new. Maximum supported API version is 1.44"}

2024/07/26 10:30:01 500 Internal Server Error - HEAD /v2/rancher/local-path-provisioner/manifests/v0.0.24?ns=docker.io - (7.489417ms) - json: cannot unmarshal object into Go value of type []main.DockerImageInfo while parsing: {"message":"client version 1.45 is too new. Maximum supported API version is 1.44"}

2024/07/26 10:30:01 500 Internal Server Error - HEAD /v2/rancher/mirrored-coredns-coredns/manifests/1.10.1?ns=docker.io - (7.607856ms) - json: cannot unmarshal object into Go value of type []main.DockerImageInfo while parsing: {"message":"client version 1.45 is too new. Maximum supported API version is 1.44"}

2024/07/26 10:30:02 404 Not Found - GET /v2/rancher/local-path-provisioner/manifests/sha256:5bb33992a4ec3034c28b5e0b3c4c2ac35d3613b25b79455eb4b1a95adc82cdc0?ns=docker.io - (42.844µs)

So, from the source code I conclude that youa re not using the Docker GO SDK? This has functionality that handles API versions.

[ligfx]

@pvannierop ah, got it. It should be fine to lower the client API version to 1.44, as I believe that corresponds with the first engine versions to export fully OCI-compliant images, 25.0.

I’ve updated the :dev tag again, can you let me know if it works?

[pvannierop]

Yes! It works now.

[ligfx]

@pvannierop great to hear it! I've pushed a v0.5 release and the v0.5 and latest tags both incorporate this change.

Fixed in d1071c3ede05f91b4826403b848342e7613f554d