search

project-zot / zot

zot - A scale-out production-ready vendor-neutral OCI-native container image/artifact registry (purely based on OCI Distribution Specification)
https://zotregistry.dev
Apache License 2.0
869 stars 93 forks source link

[Bug]: cache doesnt work, if client tries to pull a image with registry:tag@digest notation #2584

Open jkroepke opened 1 month ago

jkroepke commented 1 month ago

zot version

v2.1.0-rc2

Describe the bug

Some helm chart like ingress-nginx has both tag an digest defined in image spec. zot reports name unknown: repository name not known to registry, when configured as pull-through cache using the cache extension

To reproduce

  1. Configuration
Details ``` { "extensions": { "metrics": { "enable": true, "prometheus": { "path": "/metrics" } }, "scrub": { "enable": false }, "search": { "enable": false }, "sync": { "enable": true, "registries": [ { "content": [ { "prefix": "/bloomberg/goldpinger" } ], "onDemand": true, "tlsVerify": true, "urls": [ "https://index.docker.io" ] }, { "content": [ { "prefix": "/grafana/alloy" } ], "onDemand": true, "tlsVerify": true, "urls": [ "https://index.docker.io" ] }, { "content": [ { "prefix": "/grafana/promtail" } ], "onDemand": true, "tlsVerify": true, "urls": [ "https://index.docker.io" ] }, { "content": [ { "prefix": "/jimmidyson/configmap-reload" } ], "onDemand": true, "tlsVerify": true, "urls": [ "https://ghcr.io" ] }, { "content": [ { "prefix": "/prometheus/node-exporter" } ], "onDemand": true, "tlsVerify": true, "urls": [ "https://quay.io" ] }, { "content": [ { "prefix": "/kube-state-metrics/kube-state-metrics" } ], "onDemand": true, "tlsVerify": true, "urls": [ "https://registry.k8s.io" ] } ] }, "ui": { "enable": false } }, "http": { "address": "0.0.0.0", "port": "5000" }, "log": { "level": "debug" }, "storage": { "commit": false, "dedupe": false, "gc": true, "gcDelay": "1h", "gcInterval": "24h", "rootDirectory": "/var/lib/registry" } } ```
  1. docker pull zot/kube-state-metrics/kube-state-metrics:v2.12.0@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23e

Expected behavior

It should pull the image

Screenshots

No response

Additional context

Pull with the without digest works fine. Once the image is cached, pull with digest works.

jok@CDT-MB-20200178 ~ % docker pull zot/kube-state-metrics/kube-state-metrics:v2.12.0@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
Error response from daemon: repository zot/kube-state-metrics/kube-state-metrics not found: name unknown: repository name not known to registry
jok@CDT-MB-20200178 ~ % docker pull zot/kube-state-metrics/kube-state-metrics:v2.12.0
v2.12.0: Pulling from kube-state-metrics/kube-state-metrics
138efa92a1a8: Pull complete
e33bce57de28: Pull complete
473d8557b1b2: Pull complete
b6824ed73363: Pull complete
7c12895b777b: Pull complete
33e068de2649: Pull complete
5664b15f108b: Pull complete
27be814a09eb: Pull complete
4aa0ea1413d3: Pull complete
9ef7d74bdfdf: Pull complete
9112d77ee5b1: Pull complete
c7edb9c6f93a: Pull complete
Digest: sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
Status: Downloaded newer image for zot/kube-state-metrics/kube-state-metrics:v2.12.0
zot/kube-state-metrics/kube-state-metrics:v2.12.0
jok@CDT-MB-20200178 ~ % docker pull zot/kube-state-metrics/kube-state-metrics:v2.12.0@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe: Pulling from kube-state-metrics/kube-state-metrics
Digest: sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
Status: Image is up to date for zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
zot/kube-state-metrics/kube-state-metrics:v2.12.0@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
jok@CDT-MB-20200178 ~ % docker rmi zot/kube-state-metrics/kube-state-metrics:v2.12.0
Untagged: zot/kube-state-metrics/kube-state-metrics:v2.12.0
Untagged: zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
Deleted: sha256:6d6d84ad82b9791bf38d291dc3d5d3c8f236225b6ee00eea58af9abfc5762690
Deleted: sha256:2f014431f1cc7f94cff53a8a7d1b9d5fd5733d865b397d0132f90822ea8813f3
Deleted: sha256:61cd5a12015af0a56d9d0dddd1a6ca310b9192f4e5bb54d8fc6f580c0cd12ecd
Deleted: sha256:c58b10ae22c4696c44f1b28afea6370032de2777f3f989bc040c15efdd357c24
Deleted: sha256:1e186bfbdb3f79b6d44544623d229efdd412845752eb857cead3c2a2348ea9b8
Deleted: sha256:5a09440de7468069779da91e78341a05edea917832e3259e5ea527cb6c51e0cd
Deleted: sha256:fad5abcc062738fb16d24e26a515a85acf850c7959ddba23309b885a44f054fc
Deleted: sha256:c3ca4e08a894aeca5a2f117eb67c474bcfc902ce66dec15c15b632fa13863517
Deleted: sha256:2e38adf2638e46d3600041dbac9cd74641bb4cc47970f0d01ce84831514307bc
Deleted: sha256:0e97eff435f606822dfb6d57f8dd08402c6b0a38b0e2a403162fad225929d1ad
Deleted: sha256:824a62f32f2d2a4774ca9d0324211b44d5c5371f24574e1854785295d8af83f6
Deleted: sha256:c337e0757a51f61eab1a610f035f24bbbec932d5e0e3b913e4b8f44855c11eac
Deleted: sha256:65e963889b2fdef2798478e8b49b5cb2a6f8a4d74175703e5b329121457366c0
jok@CDT-MB-20200178 ~ % docker pull zot/kube-state-metrics/kube-state-metrics:v2.12.0@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe: Pulling from kube-state-metrics/kube-state-metrics
138efa92a1a8: Pull complete
e33bce57de28: Pull complete
473d8557b1b2: Pull complete
b6824ed73363: Pull complete
7c12895b777b: Pull complete
33e068de2649: Pull complete
5664b15f108b: Pull complete
27be814a09eb: Pull complete
4aa0ea1413d3: Pull complete
9ef7d74bdfdf: Pull complete
9112d77ee5b1: Pull complete
c7edb9c6f93a: Pull complete
Digest: sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
Status: Downloaded newer image for zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
zot/kube-state-metrics/kube-state-metrics:v2.12.0@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
rchincha commented 1 month ago

@jkroepke this is because zot is expecting a tag-based pull as the common use case. We will look at this. Thanks.

rchincha commented 1 month ago

docker pull zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe

@jkroepke can you try this instead? dropping the tag entirely?

jkroepke commented 1 month ago

Doesn't work, too.

jok@CDT-MB-20200178 ~ % docker pull zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
Error response from daemon: repository zot/kube-state-metrics/kube-state-metrics not found: name unknown: repository name not known to registry
jok@CDT-MB-20200178 ~ % docker pull zot/kube-state-metrics/kube-state-metrics:v2.12.0@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
Error response from daemon: repository zot/kube-state-metrics/kube-state-metrics not found: name unknown: repository name not known to registry
jok@CDT-MB-20200178 ~ % docker pull zot/kube-state-metrics/kube-state-metrics:v2.12.0
v2.12.0: Pulling from kube-state-metrics/kube-state-metrics
138efa92a1a8: Pull complete
e33bce57de28: Pull complete
473d8557b1b2: Pull complete
b6824ed73363: Pull complete
7c12895b777b: Pull complete
33e068de2649: Pull complete
5664b15f108b: Pull complete
27be814a09eb: Pull complete
4aa0ea1413d3: Pull complete
9ef7d74bdfdf: Pull complete
9112d77ee5b1: Pull complete
c7edb9c6f93a: Pull complete
Digest: sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
Status: Downloaded newer image for zot/kube-state-metrics/kube-state-metrics:v2.12.0
zot/kube-state-metrics/kube-state-metrics:v2.12.0
jok@CDT-MB-20200178 ~ % docker pull zot/kube-state-metrics/kube-state-metrics:v2.12.0@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe: Pulling from kube-state-metrics/kube-state-metrics
Digest: sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
Status: Image is up to date for zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
zot/kube-state-metrics/kube-state-metrics:v2.12.0@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
jok@CDT-MB-20200178 ~ % docker rmi zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
Untagged: zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
jok@CDT-MB-20200178 ~ % docker pull zot/kube-state-metrics/kube-state-metrics:v2.12.0@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe: Pulling from kube-state-metrics/kube-state-metrics
Digest: sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
Status: Downloaded newer image for zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe
zot/kube-state-metrics/kube-state-metrics:v2.12.0@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe

Once zot/kube-state-metrics/kube-state-metrics:v2.12.0 is pulled into the cache, zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe works fine. I also try docker system prune and do the pull again.

Once zot is restarted (which clears the cache), the pull does not work anymore.

andaaron commented 1 month ago

Hi @jkroepke, can you please share the sync configuration you have, and the logs produced by zot when making the call docker pull zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe

Note in the previous comment you mentioned you use docker pull zot/kube-state-metrics/kube-state-metrics:v2.12.0@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe instead of docker pull zot/kube-state-metrics/kube-state-metrics:@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe Per the OCI dist spec you either use a tag or a digest as a reference, not both: https://github.com/opencontainers/distribution-spec/blob/main/spec.md#pulling-manifests If docker uses both the tag and the digest in the same API call we don't support that. I don't know if it does, but we can check based on the zot logs.

Still this doesn't explain why the first docker pull zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe would fail. So logs should help here.

Another thing to try out is to use skopeo to request kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe, and see if that works.

jkroepke commented 1 month ago

Hi @jkroepke, can you please share the sync configuration you have

Hey @andaaron

I shared the whole zot configuration on the first post.

Another thing to try out is to use skopeo to request kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe, and see if that works.

skopeo copy docker://zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe oci-archive:image.tar.gz
FATA[0001] initializing source docker://zot/kube-state-metrics/kube-state-metrics@sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe: reading manifest sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe in zot/kube-state-metrics/kube-state-metrics: manifest unknown
jkroepke commented 1 month ago

Original Registry:

% docker pull registry.k8s.io/kube-state-metrics/kube-state-metrics@sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11
registry.k8s.io/kube-state-metrics/kube-state-metrics@sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11: Pulling from kube-state-metrics/kube-state-metrics
138efa92a1a8: Already exists
e33bce57de28: Already exists
473d8557b1b2: Already exists
b6824ed73363: Already exists
7c12895b777b: Already exists
33e068de2649: Already exists
5664b15f108b: Already exists
27be814a09eb: Already exists
4aa0ea1413d3: Already exists
9ef7d74bdfdf: Already exists
9112d77ee5b1: Already exists
c7edb9c6f93a: Already exists

zot Registry:

% docker pull zot/kube-state-metrics/kube-state-metrics@sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11
Error response from daemon: manifest for zot/kube-state-metrics/kube-state-metrics@sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11 not found: manifest unknown: manifest unknown

Per the OCI dist spec you either use a tag or a digest as a reference

You are right. Looking closer to the pull logs of the docker client, the digest will be used for pull, even both is defined.

% docker pull registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.12.0@sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11
registry.k8s.io/kube-state-metrics/kube-state-metrics@sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11: Pulling from kube-state-metrics/kube-state-metrics
andaaron commented 1 month ago

@jkroepke than you for the details. Can you please send the zot server-side logs from around the time the pull is done?

Also based on the config, it is really strange the image is no longer found when zot is restarted given storage is local. May be related to "commit=False" in the settings, but I don't remember the use case for that. @rchincha, why do we have this commit setting?

L.E. Normally the images are stored under your storage root dir /var/lib/registry/kube-state-metrics/kube-state-metrics/, and should be accessible when you restart the service

jkroepke commented 1 month ago

Also based on the config, it is really strange the image is no longer found when zot is restarted given storage is local

Thats expected, because the local storage will be deleted by Kubernetes. (emptyDir volume)

Server logs:

{"level":"info","goroutine":186,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:507","time":"2024-08-07T14:17:46.991789222Z","message":"getting available client"}
{"level":"info","remote":"https://registry.k8s.io","repository":"kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":186,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:309","time":"2024-08-07T14:17:47.005483284Z","message":"syncing image"}
{"level":"info","remote image":"registry.k8s.io/kube-state-metrics/kube-state-metrics@sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","local image":"kube-state-metrics/kube-state-metrics:sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":186,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:464","time":"2024-08-07T14:17:48.137399447Z","message":"syncing image"}
{"level":"info","syncTempDir":"/var/lib/registry/kube-state-metrics/kube-state-metrics/.sync/3c543879-2506-48c3-8d42-04dc47f3b958/kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":186,"caller":"zotregistry.dev/zot/pkg/extensions/sync/destination.go:100","time":"2024-08-07T14:17:51.451173084Z","message":"pushing synced local image to local registry"}
{"level":"error","errorType":"*errors.errorString","error":"manifest not found","dir":"/var/lib/registry/kube-state-metrics/kube-state-metrics/.sync/3c543879-2506-48c3-8d42-04dc47f3b958/kube-state-metrics/kube-state-metrics","repo":"kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":186,"caller":"zotregistry.dev/zot/pkg/extensions/sync/destination.go:108","time":"2024-08-07T14:17:51.451320518Z","message":"couldn't find synced manifest in temporary sync dir"}
{"level":"error","error":"manifest not found","errortype":"*errors.errorString","repository":"kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":186,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:485","time":"2024-08-07T14:17:51.464688328Z","message":"couldn't commit image to local image store"}
{"level":"error","error":"manifest not found","repository":"kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":184,"caller":"zotregistry.dev/zot/pkg/api/routes.go:1943","time":"2024-08-07T14:17:51.464820895Z","message":"failed to sync image"}
{"level":"error","error":": Path not found: /var/lib/registry/kube-state-metrics/kube-state-metrics/index.json","dir":"/var/lib/registry/kube-state-metrics/kube-state-metrics","goroutine":184,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1440","time":"2024-08-07T14:17:51.464969242Z","message":"failed to read index.json"}
{"level":"info","module":"http","component":"session","clientIP":"10.0.240.65:43476","method":"HEAD","path":"/v2/kube-state-metrics/kube-state-metrics/manifests/sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","statusCode":404,"latency":"5s","bodySize":281,"headers":{"User-Agent":["docker/27.1.1 go/go1.21.12 git-commit/cc13f95 kernel/6.10.0-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.1.1 \\(darwin\\))"],"X-Forwarded-For":["IP"],"X-Forwarded-Host":["zot"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"X-Forwarded-Scheme":["https"],"X-Real-Ip":["IP"],"X-Request-Id":["f36257b1e19e57c04180587010904c64"],"X-Scheme":["https"]},"goroutine":184,"caller":"zotregistry.dev/zot/pkg/api/session.go:132","time":"2024-08-07T14:17:51.465229116Z","message":"HTTP API"}
{"level":"error","error":": Path not found: /var/lib/registry/kube-state-metrics/kube-state-metrics/index.json","dir":"/var/lib/registry/kube-state-metrics/kube-state-metrics","goroutine":184,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1440","time":"2024-08-07T14:17:51.659128322Z","message":"failed to read index.json"}
{"level":"info","repository":"kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":184,"caller":"zotregistry.dev/zot/pkg/api/routes.go:1939","time":"2024-08-07T14:17:51.659217468Z","message":"trying to get updated image by syncing on demand"}
{"level":"info","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:507","time":"2024-08-07T14:17:51.659277139Z","message":"getting available client"}
{"level":"info","remote":"https://index.docker.io","repository":"kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:300","time":"2024-08-07T14:17:51.934209993Z","message":"will not sync image, filtered out by content"}
{"level":"info","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:507","time":"2024-08-07T14:17:51.934239518Z","message":"getting available client"}
{"level":"info","remote":"https://index.docker.io","repository":"kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:300","time":"2024-08-07T14:17:52.202652785Z","message":"will not sync image, filtered out by content"}
{"level":"info","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:507","time":"2024-08-07T14:17:52.202691316Z","message":"getting available client"}
{"level":"info","remote":"https://index.docker.io","repository":"kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:300","time":"2024-08-07T14:17:52.483463495Z","message":"will not sync image, filtered out by content"}
{"level":"info","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:507","time":"2024-08-07T14:17:52.483494613Z","message":"getting available client"}
{"level":"info","remote":"https://ghcr.io","repository":"kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:300","time":"2024-08-07T14:17:52.576788446Z","message":"will not sync image, filtered out by content"}
{"level":"info","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:507","time":"2024-08-07T14:17:52.576819153Z","message":"getting available client"}
{"level":"info","remote":"https://ghcr.io","repository":"kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:300","time":"2024-08-07T14:17:52.66948996Z","message":"will not sync image, filtered out by content"}
{"level":"info","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:507","time":"2024-08-07T14:17:52.669519215Z","message":"getting available client"}
{"level":"info","remote":"https://quay.io","repository":"kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:300","time":"2024-08-07T14:17:52.76465441Z","message":"will not sync image, filtered out by content"}
{"level":"info","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:507","time":"2024-08-07T14:17:52.76477247Z","message":"getting available client"}
{"level":"info","remote":"https://registry.k8s.io","repository":"kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:309","time":"2024-08-07T14:17:52.775067133Z","message":"syncing image"}
{"level":"error","error":": Path not found: /var/lib/registry/kube-state-metrics/kube-state-metrics/index.json","dir":"/var/lib/registry/kube-state-metrics/kube-state-metrics","goroutine":609,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1440","time":"2024-08-07T14:17:53.373306295Z","message":"failed to read index.json"}
{"level":"info","remote image":"registry.k8s.io/kube-state-metrics/kube-state-metrics@sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","local image":"kube-state-metrics/kube-state-metrics:sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:464","time":"2024-08-07T14:17:53.37360977Z","message":"syncing image"}
{"level":"info","syncTempDir":"/var/lib/registry/kube-state-metrics/kube-state-metrics/.sync/ef05c0e1-1a96-4102-8404-0e26522b2188/kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/destination.go:100","time":"2024-08-07T14:17:55.236490455Z","message":"pushing synced local image to local registry"}
{"level":"error","errorType":"*errors.errorString","error":"manifest not found","dir":"/var/lib/registry/kube-state-metrics/kube-state-metrics/.sync/ef05c0e1-1a96-4102-8404-0e26522b2188/kube-state-metrics/kube-state-metrics","repo":"kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/destination.go:108","time":"2024-08-07T14:17:55.236587035Z","message":"couldn't find synced manifest in temporary sync dir"}
{"level":"error","error":"manifest not found","errortype":"*errors.errorString","repository":"kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":609,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:485","time":"2024-08-07T14:17:55.249634648Z","message":"couldn't commit image to local image store"}
{"level":"error","error":"manifest not found","repository":"kube-state-metrics/kube-state-metrics","reference":"sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","goroutine":184,"caller":"zotregistry.dev/zot/pkg/api/routes.go:1943","time":"2024-08-07T14:17:55.249723273Z","message":"failed to sync image"}
{"level":"error","error":": Path not found: /var/lib/registry/kube-state-metrics/kube-state-metrics/index.json","dir":"/var/lib/registry/kube-state-metrics/kube-state-metrics","goroutine":184,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1440","time":"2024-08-07T14:17:55.249817318Z","message":"failed to read index.json"}
{"level":"info","module":"http","component":"session","clientIP":"10.0.240.65:43476","method":"GET","path":"/v2/kube-state-metrics/kube-state-metrics/manifests/sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11","statusCode":404,"latency":"3s","bodySize":242,"headers":{"Accept":["application/vnd.oci.image.index.v1+json","application/vnd.oci.image.manifest.v1+json","application/vnd.docker.distribution.manifest.v1+prettyjws","application/json","application/vnd.docker.distribution.manifest.v2+json","application/vnd.docker.distribution.manifest.list.v2+json"],"Accept-Encoding":["gzip"],"User-Agent":["docker/27.1.1 go/go1.21.12 git-commit/cc13f95 kernel/6.10.0-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.1.1 \\(darwin\\))"],"X-Forwarded-For":["IP"],"X-Forwarded-Host":["zot"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"X-Forwarded-Scheme":["https"],"X-Real-Ip":["IP"],"X-Request-Id":["cea80074863b2bb043cf82d2e29a35af"],"X-Scheme":["https"]},"goroutine":184,"caller":"zotregistry.dev/zot/pkg/api/session.go:132","time":"2024-08-07T14:17:55.249958862Z","message":"HTTP API"}

Server logs (using pull by tag):

{"level":"info","repository":"kube-state-metrics/kube-state-metrics","reference":"v2.12.0","goroutine":1469,"caller":"zotregistry.dev/zot/pkg/api/routes.go:1939","time":"2024-08-07T14:24:53.540794415Z","message":"trying to get updated image by syncing on demand"}
{"level":"info","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:507","time":"2024-08-07T14:24:53.54084004Z","message":"getting available client"}
{"level":"info","remote":"https://index.docker.io","repository":"kube-state-metrics/kube-state-metrics","reference":"v2.12.0","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:300","time":"2024-08-07T14:24:53.818301666Z","message":"will not sync image, filtered out by content"}
{"level":"info","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:507","time":"2024-08-07T14:24:53.818333033Z","message":"getting available client"}
{"level":"info","remote":"https://index.docker.io","repository":"kube-state-metrics/kube-state-metrics","reference":"v2.12.0","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:300","time":"2024-08-07T14:24:54.213290648Z","message":"will not sync image, filtered out by content"}
{"level":"info","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:507","time":"2024-08-07T14:24:54.213326813Z","message":"getting available client"}
{"level":"info","remote":"https://index.docker.io","repository":"kube-state-metrics/kube-state-metrics","reference":"v2.12.0","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:300","time":"2024-08-07T14:24:54.48277309Z","message":"will not sync image, filtered out by content"}
{"level":"info","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:507","time":"2024-08-07T14:24:54.482808725Z","message":"getting available client"}
{"level":"info","remote":"https://ghcr.io","repository":"kube-state-metrics/kube-state-metrics","reference":"v2.12.0","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:300","time":"2024-08-07T14:24:54.576205294Z","message":"will not sync image, filtered out by content"}
{"level":"info","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:507","time":"2024-08-07T14:24:54.576233906Z","message":"getting available client"}
{"level":"info","remote":"https://ghcr.io","repository":"kube-state-metrics/kube-state-metrics","reference":"v2.12.0","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:300","time":"2024-08-07T14:24:54.664231678Z","message":"will not sync image, filtered out by content"}
{"level":"info","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:507","time":"2024-08-07T14:24:54.664269948Z","message":"getting available client"}
{"level":"info","remote":"https://quay.io","repository":"kube-state-metrics/kube-state-metrics","reference":"v2.12.0","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:300","time":"2024-08-07T14:24:54.76449213Z","message":"will not sync image, filtered out by content"}
{"level":"info","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:507","time":"2024-08-07T14:24:54.764695189Z","message":"getting available client"}
{"level":"info","remote":"https://registry.k8s.io","repository":"kube-state-metrics/kube-state-metrics","reference":"v2.12.0","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:309","time":"2024-08-07T14:24:54.773966684Z","message":"syncing image"}
{"level":"error","error":": Path not found: /var/lib/registry/kube-state-metrics/kube-state-metrics/index.json","dir":"/var/lib/registry/kube-state-metrics/kube-state-metrics","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1440","time":"2024-08-07T14:24:55.423150438Z","message":"failed to read index.json"}
{"level":"info","remote image":"registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.12.0","local image":"kube-state-metrics/kube-state-metrics:v2.12.0","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:464","time":"2024-08-07T14:24:55.423552607Z","message":"syncing image"}
{"level":"info","module":"http","component":"session","clientIP":"10.0.240.53:45856","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.30"]},"goroutine":1771,"caller":"zotregistry.dev/zot/pkg/api/session.go:132","time":"2024-08-07T14:24:55.904148397Z","message":"HTTP API"}
{"level":"info","module":"http","component":"session","clientIP":"10.0.240.53:45860","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Connection":["close"],"User-Agent":["kube-probe/1.30"]},"goroutine":1772,"caller":"zotregistry.dev/zot/pkg/api/session.go:132","time":"2024-08-07T14:24:55.904405366Z","message":"HTTP API"}
{"level":"info","syncTempDir":"/var/lib/registry/kube-state-metrics/kube-state-metrics/.sync/cb0cdf2b-f770-4c00-9cf9-d6a610096479/kube-state-metrics/kube-state-metrics","reference":"v2.12.0","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/destination.go:100","time":"2024-08-07T14:24:58.150886478Z","message":"pushing synced local image to local registry"}
{"level":"warn","error":"blob not found","digest":"sha256:16100daf795a61c7588657ad2b6afc6eef7300de0c874b6c3d4def1796e28081","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.151189411Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:e33bce57de289fffd2380f73997dfb7e1ec193877904bed99f28c715d071fdc4","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.152377352Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:473d8557b1b27974f7dc7c4b4e1a209df0e27e8cae1e3e33b7bb45c969b6fc7e","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.152761517Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:b6824ed73363f94b3b2b44084c51c31bc32af77a96861d49e16f91e3ab6bed71","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.155210984Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:7c12895b777bcaa8ccae0605b4de635b68fc32d60fa08f421dc3818bf55ee212","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.155535838Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:33e068de264953dfdc9f9ada207e76b61159721fd64a4820b320d05133a55fb8","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.15589695Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:5664b15f108bf9436ce3312090a767300800edbbfd4511aa1a6d64357024d5dd","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.156223508Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:27be814a09ebd97fac6fb7b82d19f117185e90601009df3fbab6f442f85cd6b3","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.156549294Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.156862797Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:9ef7d74bdfdf3c517b28bd694a9159e94e5f53ff1ca87b39f8ca1ac0be2ed317","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.157176661Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:9112d77ee5b16873acaa186b816c3c61f5f8eba40730e729e9614a27f40211e0","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.157488652Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:9300d24220e237f370c9b8a80b5b0cb832db22d44b3b450c7b3a2a2384c7ffb7","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.158086484Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:317fd0ed16a6bf4d3760ba0c4a6e4f3caba0d605449271c98f74a0d9622c0325","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.187540877Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:facae2845703131c06d9c0875981a03285bd29445feec57fe18834e42d180130","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.190661743Z","message":"not found in cache"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/e33bce57de289fffd2380f73997dfb7e1ec193877904bed99f28c715d071fdc4","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.191288369Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/473d8557b1b27974f7dc7c4b4e1a209df0e27e8cae1e3e33b7bb45c969b6fc7e","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.19148235Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/b6824ed73363f94b3b2b44084c51c31bc32af77a96861d49e16f91e3ab6bed71","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.191688133Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/7c12895b777bcaa8ccae0605b4de635b68fc32d60fa08f421dc3818bf55ee212","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.191925104Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/33e068de264953dfdc9f9ada207e76b61159721fd64a4820b320d05133a55fb8","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.192116811Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/5664b15f108bf9436ce3312090a767300800edbbfd4511aa1a6d64357024d5dd","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.192291957Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/27be814a09ebd97fac6fb7b82d19f117185e90601009df3fbab6f442f85cd6b3","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.192472052Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.1926536Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/9ef7d74bdfdf3c517b28bd694a9159e94e5f53ff1ca87b39f8ca1ac0be2ed317","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.192847901Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/9112d77ee5b16873acaa186b816c3c61f5f8eba40730e729e9614a27f40211e0","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.193030521Z","message":"blob path found"}
{"level":"warn","error":"blob not found","digest":"sha256:ef9fe2f66beaac6275dd8fc0cab6e289cf53981e2b6f9837685981ca77309c19","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.193103076Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:8aedbaf912d4dde69c201b8378a03091d914a6b9ba34912c0e5afef15a7075b3","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.217159666Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:138efa92a1a885b92dcc9a1ceff968681fea8863e924a1f96e7eca6531c23be8","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.229693819Z","message":"not found in cache"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/e33bce57de289fffd2380f73997dfb7e1ec193877904bed99f28c715d071fdc4","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.230434507Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/473d8557b1b27974f7dc7c4b4e1a209df0e27e8cae1e3e33b7bb45c969b6fc7e","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.230693409Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/b6824ed73363f94b3b2b44084c51c31bc32af77a96861d49e16f91e3ab6bed71","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.230923477Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/7c12895b777bcaa8ccae0605b4de635b68fc32d60fa08f421dc3818bf55ee212","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.231144989Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/33e068de264953dfdc9f9ada207e76b61159721fd64a4820b320d05133a55fb8","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.23136042Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/5664b15f108bf9436ce3312090a767300800edbbfd4511aa1a6d64357024d5dd","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.231596059Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/27be814a09ebd97fac6fb7b82d19f117185e90601009df3fbab6f442f85cd6b3","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.231799297Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.232059891Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/9ef7d74bdfdf3c517b28bd694a9159e94e5f53ff1ca87b39f8ca1ac0be2ed317","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.232254544Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/9112d77ee5b16873acaa186b816c3c61f5f8eba40730e729e9614a27f40211e0","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.232479342Z","message":"blob path found"}
{"level":"warn","error":"blob not found","digest":"sha256:c7edb9c6f93acf78e11e29e1fea68c95ac82418c144c84668b5bf8cccb1f5837","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.232595428Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:6d6d84ad82b9791bf38d291dc3d5d3c8f236225b6ee00eea58af9abfc5762690","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.254796666Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:b7d3a2851213346b618885c8ea826f743b3e3f5936e4cfc1d5f7189a2b3dd07e","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.256968467Z","message":"not found in cache"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/e33bce57de289fffd2380f73997dfb7e1ec193877904bed99f28c715d071fdc4","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.257662398Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/473d8557b1b27974f7dc7c4b4e1a209df0e27e8cae1e3e33b7bb45c969b6fc7e","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.257822165Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/b6824ed73363f94b3b2b44084c51c31bc32af77a96861d49e16f91e3ab6bed71","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.257969699Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/7c12895b777bcaa8ccae0605b4de635b68fc32d60fa08f421dc3818bf55ee212","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.258124417Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/33e068de264953dfdc9f9ada207e76b61159721fd64a4820b320d05133a55fb8","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.258271912Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/5664b15f108bf9436ce3312090a767300800edbbfd4511aa1a6d64357024d5dd","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.258430076Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/27be814a09ebd97fac6fb7b82d19f117185e90601009df3fbab6f442f85cd6b3","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.25857759Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.258744411Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/9ef7d74bdfdf3c517b28bd694a9159e94e5f53ff1ca87b39f8ca1ac0be2ed317","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.259005046Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/9112d77ee5b16873acaa186b816c3c61f5f8eba40730e729e9614a27f40211e0","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.259232108Z","message":"blob path found"}
{"level":"warn","error":"blob not found","digest":"sha256:4a625dde965bce33238af0cea16d97e4c2014c6170207f1f33c2defe269e68f2","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.259352222Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:a46b6dbc044e4cb47f7af8c23c02108fdc543573d7fa56003291b6f872a1c5c1","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.281166059Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:c063d234edc556d73f33400256554556b89b29083f0803f2098bbe4553334ce6","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.283431504Z","message":"not found in cache"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/e33bce57de289fffd2380f73997dfb7e1ec193877904bed99f28c715d071fdc4","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.284070012Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/473d8557b1b27974f7dc7c4b4e1a209df0e27e8cae1e3e33b7bb45c969b6fc7e","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.284239898Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/b6824ed73363f94b3b2b44084c51c31bc32af77a96861d49e16f91e3ab6bed71","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.284494151Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/7c12895b777bcaa8ccae0605b4de635b68fc32d60fa08f421dc3818bf55ee212","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.284742593Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/33e068de264953dfdc9f9ada207e76b61159721fd64a4820b320d05133a55fb8","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.284970808Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/5664b15f108bf9436ce3312090a767300800edbbfd4511aa1a6d64357024d5dd","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.285178885Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/27be814a09ebd97fac6fb7b82d19f117185e90601009df3fbab6f442f85cd6b3","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.285399966Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.285600079Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/9ef7d74bdfdf3c517b28bd694a9159e94e5f53ff1ca87b39f8ca1ac0be2ed317","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.28583205Z","message":"blob path found"}
{"level":"debug","blob path":"/var/lib/registry/kube-state-metrics/kube-state-metrics/blobs/sha256/9112d77ee5b16873acaa186b816c3c61f5f8eba40730e729e9614a27f40211e0","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1145","time":"2024-08-07T14:24:58.286049164Z","message":"blob path found"}
{"level":"warn","error":"blob not found","digest":"sha256:3b98181601d955a2917acbc5c3a5adc5dc5221607dae63c61d51ec256037103b","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.286173285Z","message":"not found in cache"}
{"level":"warn","error":"blob not found","digest":"sha256:ac82fae6d463ee579f96413a215f8ba3254f8bb7531ae779f208b17595122222","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/storage/imagestore/imagestore.go:1161","time":"2024-08-07T14:24:58.309272945Z","message":"not found in cache"}
{"level":"info","image":"kube-state-metrics/kube-state-metrics:v2.12.0","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/destination.go:184","time":"2024-08-07T14:24:58.315197308Z","message":"successfully synced image"}
{"level":"info","component":"sync","image":"registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.12.0","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/service.go:495","time":"2024-08-07T14:24:58.328481334Z","message":"finished syncing image"}
{"level":"debug","errorType":"*errors.errorString","repository":"kube-state-metrics/kube-state-metrics","tag":"sha256-ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe.sig","error":"{\"errors\":[{\"code\":\"MANIFEST_UNKNOWN\",\"message\":\"Failed to fetch \\\"sha256-ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe.sig\\\"\"}]}\n","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/references/cosign.go:182","time":"2024-08-07T14:24:58.40211224Z","message":"couldn't find any cosign manifest for image"}
{"level":"debug","errorType":"*errors.errorString","repository":"kube-state-metrics/kube-state-metrics","tag":"sha256-ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe.sbom","error":"{\"errors\":[{\"code\":\"MANIFEST_UNKNOWN\",\"message\":\"Failed to fetch \\\"sha256-ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe.sbom\\\"\"}]}\n","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/references/cosign.go:182","time":"2024-08-07T14:24:58.445370747Z","message":"couldn't find any cosign manifest for image"}
{"level":"debug","repository":"kube-state-metrics/kube-state-metrics","subject":"sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/references/referrers_tag.go:159","time":"2024-08-07T14:24:58.492080378Z","message":"couldn't find any index with referrers tag for image, skipping"}
{"level":"debug","error":"couldn't find upstream referrer","reference type":"TagReference","image":"kube-state-metrics/kube-state-metrics:sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/references/references.go:105","time":"2024-08-07T14:24:58.49215689Z","message":"couldn't sync image referrer"}
{"level":"debug","repository":"kube-state-metrics/kube-state-metrics","subject":"sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/references/oci.go:167","time":"2024-08-07T14:24:58.595209078Z","message":"couldn't find any oci reference for image, skipping"}
{"level":"debug","error":"couldn't find upstream referrer","reference type":"OCIReference","image":"kube-state-metrics/kube-state-metrics:sha256:ca0b12d75f673fa03f9a69aa30fa904c52525251395a8e7d813f23eca7a3dbfe","goroutine":1471,"caller":"zotregistry.dev/zot/pkg/extensions/sync/references/references.go:105","time":"2024-08-07T14:24:58.595255645Z","message":"couldn't sync image referrer"}
{"level":"info","module":"http","component":"session","clientIP":"10.0.240.65:51110","method":"HEAD","path":"/v2/kube-state-metrics/kube-state-metrics/manifests/v2.12.0","statusCode":200,"latency":"5s","bodySize":0,"headers":{"Accept":["application/vnd.docker.distribution.manifest.v2+json","application/vnd.docker.distribution.manifest.list.v2+json","application/vnd.oci.image.index.v1+json","application/vnd.oci.image.manifest.v1+json","application/vnd.docker.distribution.manifest.v1+prettyjws","application/json"],"User-Agent":["docker/27.1.1 go/go1.21.12 git-commit/cc13f95 kernel/6.10.0-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/27.1.1 \\(darwin\\))"],"X-Forwarded-For":["IP"],"X-Forwarded-Host":["zot"],"X-Forwarded-Port":["443"],"X-Forwarded-Proto":["https"],"X-Forwarded-Scheme":["https"],"X-Real-Ip":["IP"],"X-Request-Id":["05c6be88b152990e6bc57f501be9e915"],"X-Scheme":["https"]},"goroutine":1469,"caller":"zotregistry.dev/zot/pkg/api/session.go:132","time":"2024-08-07T14:24:58.595553599Z","message":"HTTP API"}
andaaron commented 1 month ago

Thats expected, because the local storage will be deleted by Kubernetes. (emptyDir volume)

OK, I assumed you wanted the images to be available in zot storage after a zot restart.

jkroepke commented 1 month ago

OK, I assumed you wanted the images to be available in zot storage after a zot restart.

Yeah, thats why I'm working on #2223

andaaron commented 1 month ago

Based on the logs it should have been copied to a local temporary folder ("/var/lib/registry/kube-state-metrics/kube-state-metrics/.sync/ef05c0e1-1a96-4102-8404-0e26522b2188/kube-state-metrics/kube-state-metrics"), but in the next function call it doesn't find it there.

We need to investigate this further.

andaaron commented 1 month ago

I did some digging, reproduced, and I think I found the root cause:

regctl manifest get registry.k8s.io/kube-state-metrics/kube-state-metrics@sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11
Name:        registry.k8s.io/kube-state-metrics/kube-state-metrics@sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11
MediaType:   application/vnd.docker.distribution.manifest.list.v2+json
Digest:      sha256:b401fae262a5decf83c4311083f8efb4d6ca7b6a733e57b95344cb8dccd14e11

Manifests:

  Name:      registry.k8s.io/kube-state-metrics/kube-state-metrics@sha256:af36e32f56c6f2e1f861525e8c438d6913ea236b5480aafe4bc6ab5d4b9d40f1
  Digest:    sha256:af36e32f56c6f2e1f861525e8c438d6913ea236b5480aafe4bc6ab5d4b9d40f1
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      registry.k8s.io/kube-state-metrics/kube-state-metrics@sha256:7016366b2851f6330e100bcd7d6b3feed64d7befe5d6d58d0bdebdf518e26f04
  Digest:    sha256:7016366b2851f6330e100bcd7d6b3feed64d7befe5d6d58d0bdebdf518e26f04
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v7

  Name:      registry.k8s.io/kube-state-metrics/kube-state-metrics@sha256:0b849c2a1fd2a8e3a8465803c0e0853e063fd89b33ad10847f82152b8f1f7066
  Digest:    sha256:0b849c2a1fd2a8e3a8465803c0e0853e063fd89b33ad10847f82152b8f1f7066
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64

  Name:      registry.k8s.io/kube-state-metrics/kube-state-metrics@sha256:cfb944b712b379bad10a9f5f9346f8f793f9fea991b020a3912653116f86179e
  Digest:    sha256:cfb944b712b379bad10a9f5f9346f8f793f9fea991b020a3912653116f86179e
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/ppc64le

  Name:      registry.k8s.io/kube-state-metrics/kube-state-metrics@sha256:2aa7a435eacb32c5a301ccdeba8bb8d753c2eb438e4708c8addc4a736303bf58
  Digest:    sha256:2aa7a435eacb32c5a301ccdeba8bb8d753c2eb438e4708c8addc4a736303bf58
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/s390x

This is a docker media type: application/vnd.docker.distribution.manifest.list.v2+json which we don't support internally in zot. This media type gets converted automatically to an oci media type, and since the format/content is different, the digest is different. When zot tries to find the image locally using the old docker digest, it fails.

andaaron commented 1 month ago

I think we could add a "fix" in zot to return the files in OCI format to the client, but they wouldn't have the same digests, and if the local docker does any validation on the response body digests it would find the digest mismatch.

rchincha commented 1 month ago

@rchincha, why do we have this commit setting?

This setting makes sure that data is actually persisted on the disk before responding back to the client, else it could just be in un-flushed buffers which is faster but exposes to data losses.

jkroepke commented 1 month ago

but they wouldn't have the same digests, and if the local docker does any validation on the response body digests it would find the digest mismatch.

Not sure, if that an good approach. I guess it would also break signatures and provenance?

skoepo has an option to preserve digests (https://github.com/containers/skopeo/blob/5af7b5dba12ce9a157e96e030d477d5946e5fb27/cmd/skopeo/copy.go#L87)

In background, it using the flag provide by github.com/containers/image

Why it cant be used at zot?

andaaron commented 1 month ago

As far as I can tell that is an option to fail the operation in case the digests cannot be preserved. And digests can't be the same since the content of the file is different, it's a different json struct.

You are right about signatures and other referrers. See also https://github.com/project-zot/zot/issues/724