Open flyfax opened 1 month ago
brandond
commented
1 month ago You're using $1 in your rewrite but do not have a capture group in the regex so this will not be filled with anything. What exactly are you trying to do? Please check the docs for examples.
flyfax
commented
1 month ago I'm trying to let the pod pull image from the mirror registry docker-na-public.artifactory.test.com/se-next-gen-docker-local/mini-test@sha256:c718d3f996061aef92966a2171713af1cfdbac93cbea7a753107e3d5430c3687 instead of original image definition cp.icr.io/cp/se-data-center-edge/mini-test@sha256:c718d3f996061aef92966a2171713af1cfdbac93cbea7a753107e3d5430c3687
using rewrite part to replace cp/se-data-center-edge to se-next-gen-docker-local
I look at the example here: https://docs.k3s.io/installation/private-registry. And also try to both configuration in rewrite part
rewrite:
"cp/se-data-center-edge": "se-next-gen-docker-local/$1"and
rewrite:
"cp/se-data-center-edge/(.*)": "se-next-gen-docker-local/$1"But I got the same error which seems rewrite part does not effect it.
brandond
commented
1 month ago Can you confirm that you're not using a custom containerd.toml.tmpl?
Also, verify the contents of /var/lib/rancher/k3s/agent/etc/containerd/certs.d/cp.icr.io/hosts.toml - do you see the rewrite in there?
You might also check the containerd logs to see if it contains any interesting errors regarding the pull.
brandond
commented
1 month ago
failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://docker-na-public.artifactory.test.com/artifactory/api/docker/null/v2/token?scope=repository%3Acp%2Fse-data--center-edge%2Fmini-test%3Apull&service=docker-na-public.artifactory.test.com: 401 Unauthorized
https://docker-na-public.artifactory.test.com/artifactory/api/docker/null/v2/token
The null in this URL looks weird. Are you still getting that after fixing the regex?
The message also suggests that there is an extra hyphen coming from somewhere... the scope is repository:cp/se-data--center-edge/mini-test:pull which does not match what you said you're trying to pull. Did you perhaps typo the image in your pod spec as cp.icr.io/cp/se-data--center-edge/mini-test:latest, or add an extra hyphen in your replacement string?
flyfax
commented
1 month ago I did not use containerd.toml.tmpl, and rewrite part is in the host
[root@qb-reg5-m1 containerd]# ls
certs.d config.toml
[root@qb-reg5-m1 containerd]# pwd
/var/lib/rancher/k3s/agent/etc/containerd
[root@qb-reg5-m1 containerd]# cat certs.d/cp.icr.io/hosts.toml
# File generated by k3s. DO NOT EDIT.
server = "https://cp.icr.io/v2"
capabilities = ["pull", "resolve", "push"]
[host]
[host."https://docker-na-public.artifactory.test.com/v2"]
capabilities = ["pull", "resolve"]
[host."https://docker-na-public.artifactory.test.com/v2".rewrite]
"cp/se-data-center-edge/(.*)" = "se-next-gen-docker-local/$1"
The message also suggests that there is an extra hyphen coming from somewhere... the scope is
repository:cp/se-data--center-edge/mini-test:pullwhich does not match what you said you're trying to pull. Did you perhaps typo the image in your pod spec ascp.icr.io/cp/se-data--center-edge/mini-test:latest, or add an extra hyphen in your replacement string?
Yes, I still get the same error after fixing the regex.
The interesting thing is the first registry mirror working well.
I could pull image from docker-na-public.artifactory.test.com/se-next-gen-docker-local/edge-operator-catalog@sha256:4f9725b23c8560eae25be0a9fac01c74c9d4a9fee8200e31aad9842f7c338433 instead of original path
icr.io/cpopen/edge-operator-catalog@sha256:4f9725b23c8560eae25be0a9fac01c74c9d4a9fee8200e31aad9842f7c338433
Not sure if the issue is because of registry name 'cp.icr.io' includes 'cp' which part of regex?
brandond
commented
1 month ago It occurs to me - you've got registries.yaml on BOTH the nodes, right? That is node-specific configuration; it is not global cluster config. You need to configure that on the agent AND the server individually.
Assuming you've don that, You might try doing the following on whatever node the pod is being pulled from:
echo CONTAINERD_LOG_LEVEL=debug >> /etc/sysconfig/k3s && systemctl restart k3s (on a server)
echo CONTAINERD_LOG_LEVEL=debug >> /etc/sysconfig/k3s-agent && systemctl restart k3s-agent (on an agent)
That'll give you more info in the containerd.log
flyfax
commented
1 month ago Yes, I put registries.yaml in both server and agent nodes. Thanks for the suggestion, I will try to enable debug to see how it looks.
brandond
commented
1 month ago Just on the off chance the replacement is doing something weird, you might also try anchoring it?
rewrite:
"^cp/se-data-center-edge/(.+)$": "se-next-gen-docker-local/$1"
codering
commented
4 weeks ago I have similar problem.
k3s version v1.29.4+k3s1 (94e29e2e)
go version go1.21.9I cannot access Docker Hub, so I have placed the images on my own registry.
/etc/rancher/k3s/registries.yaml
mirrors:
"docker.io":
endpoint:
- https://swr.cn-east-3.myhuaweicloud.com
rewrite:
"(.*)": "hmirror/$1"
configs:
swr.cn-east-3.myhuaweicloud.com:
auth:
username: xx
password: yyInstall k3s on a single node
curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_SKIP_SELINUX_RPM=true K3S_KUBECONFIG_MODE="644" INSTALL_K3S_MIRROR=cn K3S_TOKEN=SECRET INSTALL_K3S_VERSION="v1.29.4+k3s1" sh -I can see images have pulled normally from my registry when k3s install.
[root@ecs-free-0001 tmp]# crictl images
IMAGE TAG IMAGE ID SIZE
docker.io/rancher/klipper-helm v0.8.3-build20240228 0929b4140ada6 91.2MB
docker.io/rancher/klipper-lb v0.4.7 edc812b8e25d0 4.78MB
docker.io/rancher/local-path-provisioner v0.0.26 c54dcef6214cb 17.2MB
docker.io/rancher/mirrored-coredns-coredns 1.10.1 ead0a4a53df89 16.2MB
docker.io/rancher/mirrored-library-traefik 2.10.7 ee69e8120b64a 43.2MB
docker.io/rancher/mirrored-metrics-server v0.7.0 b9a5a1927366a 19.3MB
docker.io/rancher/mirrored-pause 3.6 6270bb605e12e 298kB[root@ecs-free-0001 tmp]# kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system local-path-provisioner-6c86858495-m7p9f 1/1 Running 0 15m
kube-system svclb-traefik-839f5d4c-rkz2c 2/2 Running 0 12m
kube-system helm-install-traefik-crd-tssm4 0/1 Completed 0 15m
kube-system helm-install-traefik-frdwz 0/1 Completed 1 15m
kube-system coredns-6799fbcd5-9z2gm 1/1 Running 0 15m
kube-system traefik-7d5f6474df-kfzgh 1/1 Running 0 12m
kube-system metrics-server-54fd9b65b-fd5nn 1/1 Running 0 15m
when I pull another one image with original url from my registry , it's OK.
[root@ecs-free-0001 ~]# crictl pull swr.cn-east-3.myhuaweicloud.com/hmirror/rabbitmqoperator/cluster-operator:2.8.0
Image is up to date for sha256:c0a9306b27689ddde5429e1333bac7b5ca9dc49cf005918a49518fbebbfd9d8b
[root@ecs-free-0001 ~]# crictl images | grep cluster-operator
swr.cn-east-3.myhuaweicloud.com/hmirror/rabbitmqoperator/cluster-operator 2.8.0 c0a9306b27689 26MB
[root@ecs-free-0001 ~]#but I can't pull it with rewrite. I don't know why.
[root@ecs-free-0001 tmp]# crictl pull rabbitmqoperator/cluster-operator:2.8.0
E1101 17:31:20.620215 16360 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"docker.io/rabbitmqoperator/cluster-operator:2.8.0\": failed to resolve reference \"docker.io/rabbitmqoperator/cluster-operator:2.8.0\": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/?scope=repository%3Ahmirror%2Frabbitmqoperator%2Fcluster-operator%3A&scope=repository%3Arabbitmqoperator%2Fcluster-operator%3Apull&service=dockyard: 404 Not Found" image="rabbitmqoperator/cluster-operator:2.8.0"
FATA[0000] pulling image: failed to pull and unpack image "docker.io/rabbitmqoperator/cluster-operator:2.8.0": failed to resolve reference "docker.io/rabbitmqoperator/cluster-operator:2.8.0": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/?scope=repository%3Ahmirror%2Frabbitmqoperator%2Fcluster-operator%3A&scope=repository%3Arabbitmqoperator%2Fcluster-operator%3Apull&service=dockyard: 404 Not FoundDid I make a mistake in my configuration somewhere? But why is it able to normally pull the rancher images during the k3s installation?
brandond
commented
4 weeks ago @codering this looks like an issue with the registry you're using as a mirror. I don't know why it would be returning a 404 when you're authenticating to use it as a mirror. I do see that the auth request has the scope set twice, once for the original image repo, and once with the rewritten name. I don't think I've seen this before, but I suspect this is confusing the auth service. You might turn on containerd debug and trace the requests to see where this is coming from.
flyfax
commented
3 weeks ago @brandond After modifying regex in rewrite part, I could pull image from both server and agent using 'crictl'
#crictl pull cp.icr.io/cp/se-data--center-edge/mini-test:@sha256:ca9a0906f7810d2a827648960ee2c6a8c9980817474ce30eee546e19d7f78132
Image is up to date for sha256:2f6e40f487db28d0d728d3f7c05248edd7a270a8c72726e9ef311d718c8f2dde
But I don't understand why image pull inside the pod still failed. Do you have any other suggestions? Thank you.
Warning Failed 3m38s (x4 over 5m10s) kubelet Failed to pull image "cp.icr.io/cp/se-data-center-edge/mini-test:@sha256:ca9a0906f7810d2a827648960ee2c6a8c9980817474ce30eee546e19d7f78132": failed to pull and unpack image "cp.icr.io/cp/se-data-center-edge/mini-test:@sha256:ca9a0906f7810d2a827648960ee2c6a8c9980817474ce30eee546e19d7f78132": failed to resolve reference "cp.icr.io/cp/se-data-center-edge/mini-test:@sha256:ca9a0906f7810d2a827648960ee2c6a8c9980817474ce30eee546e19d7f78132": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://docker-na-public.artifactory.swg-devops.com/artifactory/api/docker/null/v2/token?scope=repository%3Acp%2Fse-data-center-edge%2Fmini-test%3Apull&service=docker-na-public.artifactory.swg-devops.com: 401 UnauthorizedOne of these has a double hyphen and the other does not. I point this out earlier. Is this intentional?
codering
commented
3 weeks ago @codering this looks like an issue with the registry you're using as a mirror. I don't know why it would be returning a 404 when you're authenticating to use it as a mirror. I do see that the auth request has the scope set twice, once for the original image repo, and once with the rewritten name. I don't think I've seen this before, but I suspect this is confusing the auth service. You might turn on containerd debug and trace the requests to see where this is coming from.
@brandond Set containerd debug level to debug. Here are the detail logs. Any ideas on the issue?
crictl pull docker.io/rabbitmqoperator/cluster-operator:2.7.0time="2024-11-04T09:23:59.282048240+08:00" level=info msg="PullImage \"docker.io/rabbitmqoperator/cluster-operator:2.7.0\""
time="2024-11-04T09:23:59.282126759+08:00" level=debug msg="PullImage \"docker.io/rabbitmqoperator/cluster-operator:2.7.0\" with snapshotter overlayfs"
time="2024-11-04T09:23:59.283737151+08:00" level=debug msg="loading host directory" dir=/var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io
time="2024-11-04T09:23:59.283970296+08:00" level=debug msg=resolving host=swr.cn-east-3.myhuaweicloud.com
time="2024-11-04T09:23:59.283997031+08:00" level=debug msg="do request" host=swr.cn-east-3.myhuaweicloud.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.7.15-k3s1 request.method=HEAD url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/2.7.0?ns=docker.io"
time="2024-11-04T09:23:59.324497576+08:00" level=debug msg="fetch response received" host=swr.cn-east-3.myhuaweicloud.com response.header.connection=keep-alive response.header.content-length=61 response.header.content-type="application/json;charset=UTF-8" response.header.date="Mon, 04 Nov 2024 01:23:59 GMT" response.header.forserver=swr response.header.keep-alive="timeout=300" response.header.server="Web Server" response.header.www-authenticate="Bearer realm=\"https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/\",service=\"dockyard\",scope=\"repository:hmirror/rabbitmqoperator/cluster-operator:\"" response.status="401 Unauthorized" url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/2.7.0?ns=docker.io"
time="2024-11-04T09:23:59.324542691+08:00" level=debug msg=Unauthorized header="Bearer realm=\"https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/\",service=\"dockyard\",scope=\"repository:hmirror/rabbitmqoperator/cluster-operator:\"" host=swr.cn-east-3.myhuaweicloud.com
time="2024-11-04T09:23:59.324578114+08:00" level=debug msg="do request" host=swr.cn-east-3.myhuaweicloud.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.7.15-k3s1 request.method=HEAD url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/2.7.0?ns=docker.io"
time="2024-11-04T09:23:59.617911572+08:00" level=info msg="trying next host" error="failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/?scope=repository%3Ahmirror%2Frabbitmqoperator%2Fcluster-operator%3A&scope=repository%3Arabbitmqoperator%2Fcluster-operator%3Apull&service=dockyard: 404 Not Found" host=swr.cn-east-3.myhuaweicloud.com
time="2024-11-04T09:23:59.617949396+08:00" level=debug msg=resolving host=registry-1.docker.io
time="2024-11-04T09:23:59.617966636+08:00" level=debug msg="do request" host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.7.15-k3s1 request.method=HEAD url="https://registry-1.docker.io/v2/rabbitmqoperator/cluster-operator/manifests/2.7.0"
time="2024-11-04T09:23:59.829470795+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://registry-1.docker.io/v2/rabbitmqoperator/cluster-operator/manifests/2.7.0\": read tcp 192.168.0.101:59858->54.236.113.205:443: read: connection reset by peer" host=registry-1.docker.io
time="2024-11-04T09:23:59.830960725+08:00" level=error msg="PullImage \"docker.io/rabbitmqoperator/cluster-operator:2.7.0\" failed" error="failed to pull and unpack image \"docker.io/rabbitmqoperator/cluster-operator:2.7.0\": failed to resolve reference \"docker.io/rabbitmqoperator/cluster-operator:2.7.0\": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/?scope=repository%3Ahmirror%2Frabbitmqoperator%2Fcluster-operator%3A&scope=repository%3Arabbitmqoperator%2Fcluster-operator%3Apull&service=dockyard: 404 Not Found"
time="2024-11-04T09:23:59.831067745+08:00" level=info msg="stop pulling image docker.io/rabbitmqoperator/cluster-operator:2.7.0: active requests=0, bytes read=194"
crictl pull swr.cn-east-3.myhuaweicloud.com/hmirror/rabbitmqoperator/cluster-operator:2.7.0time="2024-11-04T09:40:51.926426596+08:00" level=info msg="PullImage \"swr.cn-east-3.myhuaweicloud.com/hmirror/rabbitmqoperator/cluster-operator:2.7.0\""
time="2024-11-04T09:40:51.926489223+08:00" level=debug msg="PullImage \"swr.cn-east-3.myhuaweicloud.com/hmirror/rabbitmqoperator/cluster-operator:2.7.0\" with snapshotter overlayfs"
time="2024-11-04T09:40:51.927968493+08:00" level=debug msg="loading host directory" dir=/var/lib/rancher/k3s/agent/etc/containerd/certs.d/swr.cn-east-3.myhuaweicloud.com
time="2024-11-04T09:40:51.928107089+08:00" level=debug msg=resolving host=swr.cn-east-3.myhuaweicloud.com
time="2024-11-04T09:40:51.928129911+08:00" level=debug msg="do request" host=swr.cn-east-3.myhuaweicloud.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.7.15-k3s1 request.method=HEAD url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/2.7.0"
time="2024-11-04T09:40:51.955075769+08:00" level=debug msg="fetch response received" host=swr.cn-east-3.myhuaweicloud.com response.header.connection=keep-alive response.header.content-length=61 response.header.content-type="application/json;charset=UTF-8" response.header.date="Mon, 04 Nov 2024 01:40:52 GMT" response.header.forserver=swr response.header.keep-alive="timeout=300" response.header.server="Web Server" response.header.www-authenticate="Bearer realm=\"https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/\",service=\"dockyard\",scope=\"repository:hmirror/rabbitmqoperator/cluster-operator:\"" response.status="401 Unauthorized" url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/2.7.0"
time="2024-11-04T09:40:51.955120497+08:00" level=debug msg=Unauthorized header="Bearer realm=\"https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/\",service=\"dockyard\",scope=\"repository:hmirror/rabbitmqoperator/cluster-operator:\"" host=swr.cn-east-3.myhuaweicloud.com
time="2024-11-04T09:40:51.955151765+08:00" level=debug msg="do request" host=swr.cn-east-3.myhuaweicloud.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.7.15-k3s1 request.method=HEAD url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/2.7.0"
time="2024-11-04T09:40:52.458440519+08:00" level=debug msg="fetch response received" host=swr.cn-east-3.myhuaweicloud.com response.header.cache-control="no-cache, no-store, must-revalidate" response.header.connection=keep-alive response.header.content-length=946 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Mon, 04 Nov 2024 01:40:52 GMT" response.header.docker-content-digest="sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c" response.header.forserver=swr response.header.keep-alive="timeout=300" response.header.pragma=no-cache response.header.record-context-id=d91caf1a-e647-405f-9bc3-ed7cac40a09f response.header.server="Web Server" response.header.strict-transport-security="max-age=31536000; includeSubdomains;" response.header.x-content-type-options=nosniff response.header.x-download-options=noopen response.header.x-xss-protection="1; mode=block" response.status="200 OK" url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/2.7.0"
time="2024-11-04T09:40:52.458494997+08:00" level=debug msg=resolved desc.digest="sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c" host=swr.cn-east-3.myhuaweicloud.com
time="2024-11-04T09:40:52.458534621+08:00" level=debug msg="loading host directory" dir=/var/lib/rancher/k3s/agent/etc/containerd/certs.d/swr.cn-east-3.myhuaweicloud.com
time="2024-11-04T09:40:52.458746900+08:00" level=debug msg=fetch digest="sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c" mediatype=application/vnd.docker.distribution.manifest.v2+json size=946
time="2024-11-04T09:40:52.464990700+08:00" level=debug msg="do request" digest="sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/v1.7.15-k3s1 request.method=GET size=946 url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c"
time="2024-11-04T09:40:52.496004396+08:00" level=debug msg="fetch response received" digest="sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.connection=keep-alive response.header.content-length=61 response.header.content-type="application/json;charset=UTF-8" response.header.date="Mon, 04 Nov 2024 01:40:52 GMT" response.header.forserver=swr response.header.keep-alive="timeout=300" response.header.server="Web Server" response.header.www-authenticate="Bearer realm=\"https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/\",service=\"dockyard\",scope=\"repository:hmirror/rabbitmqoperator/cluster-operator:pull\"" response.status="401 Unauthorized" size=946 url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c"
time="2024-11-04T09:40:52.496056098+08:00" level=debug msg=Unauthorized digest="sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c" header="Bearer realm=\"https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/\",service=\"dockyard\",scope=\"repository:hmirror/rabbitmqoperator/cluster-operator:pull\"" mediatype=application/vnd.docker.distribution.manifest.v2+json size=946
time="2024-11-04T09:40:52.496125955+08:00" level=debug msg="do request" digest="sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/v1.7.15-k3s1 request.method=GET size=946 url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c"
time="2024-11-04T09:40:52.732902088+08:00" level=debug msg="fetch response received" digest="sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.cache-control="no-cache, no-store, must-revalidate" response.header.connection=keep-alive response.header.content-length=946 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Mon, 04 Nov 2024 01:40:53 GMT" response.header.docker-content-digest="sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c" response.header.forserver=swr response.header.keep-alive="timeout=300" response.header.pragma=no-cache response.header.record-context-id=a76b6c60-08df-4a8e-99e8-659062d1d4f4 response.header.server="Web Server" response.header.strict-transport-security="max-age=31536000; includeSubdomains;" response.header.x-content-type-options=nosniff response.header.x-download-options=noopen response.header.x-xss-protection="1; mode=block" response.status="200 OK" size=946 url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c"
time="2024-11-04T09:40:52.749338863+08:00" level=debug msg=fetch digest="sha256:2324be6c5743ad72bedb72bec9beaf989bd5bace9ed49992566ee2bcda03dcdd" mediatype=application/vnd.docker.container.image.v1+json size=2169
time="2024-11-04T09:40:52.755289897+08:00" level=debug msg="do request" digest="sha256:2324be6c5743ad72bedb72bec9beaf989bd5bace9ed49992566ee2bcda03dcdd" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/v1.7.15-k3s1 request.method=GET size=2169 url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/blobs/sha256:2324be6c5743ad72bedb72bec9beaf989bd5bace9ed49992566ee2bcda03dcdd"
time="2024-11-04T09:40:52.937208009+08:00" level=debug msg="fetch response received" digest="sha256:2324be6c5743ad72bedb72bec9beaf989bd5bace9ed49992566ee2bcda03dcdd" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.connection=keep-alive response.header.content-length=2169 response.header.content-type=binary/octet-stream response.header.date="Mon, 04 Nov 2024 01:40:53 GMT" response.header.etag="\"501498a4e7f27eb87471ad6614a88204\"" response.header.last-modified="Mon, 17 Jun 2024 10:23:51 GMT" response.header.server=OBS response.header.x-amz-id-2=32AAAQAAEAABAAAQAAEAABAAAQAAEAABCSAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA response.header.x-amz-request-id=00000192F4D50189B924EB46C4172C71 response.header.x-amz-tagging-count=0 response.header.x-reserved="amazon, aws and amazon web services are trademarks or registered trademarks of Amazon Technologies, Inc" response.status="200 OK" size=2169 url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/blobs/sha256:2324be6c5743ad72bedb72bec9beaf989bd5bace9ed49992566ee2bcda03dcdd"
time="2024-11-04T09:40:52.948956200+08:00" level=debug msg=fetch digest="sha256:f64c8f418cb03a7c30ffe585d63beddccadbae2468a802f6c59ad713e15d307f" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=123555
time="2024-11-04T09:40:52.949006150+08:00" level=debug msg=fetch digest="sha256:315d362218bb7f6cb1d3ad08799c92a30e7dc43e48a1fc8cee05c38e247a9138" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=25836762
time="2024-11-04T09:40:52.949027721+08:00" level=debug msg=fetch digest="sha256:ed3710ae7c65e8f0f4ef30639ff831de697979201507848469a341c1a1e9e1d7" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=219
time="2024-11-04T09:40:52.955347530+08:00" level=debug msg="do request" digest="sha256:f64c8f418cb03a7c30ffe585d63beddccadbae2468a802f6c59ad713e15d307f" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/v1.7.15-k3s1 request.method=GET size=123555 url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/blobs/sha256:f64c8f418cb03a7c30ffe585d63beddccadbae2468a802f6c59ad713e15d307f"
time="2024-11-04T09:40:52.963426729+08:00" level=debug msg="do request" digest="sha256:315d362218bb7f6cb1d3ad08799c92a30e7dc43e48a1fc8cee05c38e247a9138" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/v1.7.15-k3s1 request.method=GET size=25836762 url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/blobs/sha256:315d362218bb7f6cb1d3ad08799c92a30e7dc43e48a1fc8cee05c38e247a9138"
time="2024-11-04T09:40:52.968779188+08:00" level=debug msg="do request" digest="sha256:ed3710ae7c65e8f0f4ef30639ff831de697979201507848469a341c1a1e9e1d7" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/v1.7.15-k3s1 request.method=GET size=219 url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/blobs/sha256:ed3710ae7c65e8f0f4ef30639ff831de697979201507848469a341c1a1e9e1d7"
time="2024-11-04T09:40:53.062104025+08:00" level=debug msg="fetch response received" digest="sha256:f64c8f418cb03a7c30ffe585d63beddccadbae2468a802f6c59ad713e15d307f" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.connection=keep-alive response.header.content-length=123555 response.header.content-type=binary/octet-stream response.header.date="Mon, 04 Nov 2024 01:40:53 GMT" response.header.etag="\"52ea2d3679acc6544bb93e694f9ca78b\"" response.header.last-modified="Mon, 17 Jun 2024 10:23:04 GMT" response.header.server=OBS response.header.x-amz-id-2=32AAAQAAEAABAAAQAAEAABAAAQAAEAABCSAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA response.header.x-amz-request-id=00000192F4D50213B924EB46C4172C73 response.header.x-amz-tagging-count=0 response.header.x-reserved="amazon, aws and amazon web services are trademarks or registered trademarks of Amazon Technologies, Inc" response.status="200 OK" size=123555 url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/blobs/sha256:f64c8f418cb03a7c30ffe585d63beddccadbae2468a802f6c59ad713e15d307f"
time="2024-11-04T09:40:53.194398712+08:00" level=debug msg="fetch response received" digest="sha256:315d362218bb7f6cb1d3ad08799c92a30e7dc43e48a1fc8cee05c38e247a9138" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.connection=keep-alive response.header.content-length=25836762 response.header.content-type=binary/octet-stream response.header.date="Mon, 04 Nov 2024 01:40:53 GMT" response.header.etag="\"718af2a39601292680b52d232986eddc\"" response.header.last-modified="Mon, 17 Jun 2024 10:23:47 GMT" response.header.server=OBS response.header.x-amz-id-2=32AAAQAAEAABAAAQAAEAABAAAQAAEAABCSAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA response.header.x-amz-request-id=00000192F4D50275B664641F9D9F8BBF response.header.x-amz-tagging-count=0 response.header.x-reserved="amazon, aws and amazon web services are trademarks or registered trademarks of Amazon Technologies, Inc" response.status="200 OK" size=25836762 url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/blobs/sha256:315d362218bb7f6cb1d3ad08799c92a30e7dc43e48a1fc8cee05c38e247a9138"
time="2024-11-04T09:40:53.834965790+08:00" level=debug msg="diff applied" d=312.014418ms digest="sha256:315d362218bb7f6cb1d3ad08799c92a30e7dc43e48a1fc8cee05c38e247a9138" media=application/vnd.docker.image.rootfs.diff.tar.gzip size=25836762
time="2024-11-04T09:40:53.841669479+08:00" level=debug msg="layer unpacked" duration=898.965785ms layer="sha256:315d362218bb7f6cb1d3ad08799c92a30e7dc43e48a1fc8cee05c38e247a9138"
time="2024-11-04T09:40:54.206225497+08:00" level=debug msg="fetch response received" digest="sha256:ed3710ae7c65e8f0f4ef30639ff831de697979201507848469a341c1a1e9e1d7" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.connection=keep-alive response.header.content-length=219 response.header.content-type=binary/octet-stream response.header.date="Mon, 04 Nov 2024 01:40:54 GMT" response.header.etag="\"99e2a9f018cd5e0a7912c811ab1112cf\"" response.header.last-modified="Mon, 17 Jun 2024 10:23:05 GMT" response.header.server=OBS response.header.x-amz-id-2=32AAAQAAEAABAAAQAAEAABAAAQAAEAABCSAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA response.header.x-amz-request-id=00000192F4D5027AB16588C7EC5058ED response.header.x-amz-tagging-count=0 response.header.x-reserved="amazon, aws and amazon web services are trademarks or registered trademarks of Amazon Technologies, Inc" response.status="200 OK" size=219 url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/blobs/sha256:ed3710ae7c65e8f0f4ef30639ff831de697979201507848469a341c1a1e9e1d7"
time="2024-11-04T09:40:54.213011589+08:00" level=debug msg="diff applied" d="908.773µs" digest="sha256:ed3710ae7c65e8f0f4ef30639ff831de697979201507848469a341c1a1e9e1d7" media=application/vnd.docker.image.rootfs.diff.tar.gzip size=219
time="2024-11-04T09:40:54.217570977+08:00" level=debug msg="layer unpacked" duration=375.869091ms layer="sha256:ed3710ae7c65e8f0f4ef30639ff831de697979201507848469a341c1a1e9e1d7"
time="2024-11-04T09:40:54.223826732+08:00" level=debug msg="diff applied" d=1.954618ms digest="sha256:f64c8f418cb03a7c30ffe585d63beddccadbae2468a802f6c59ad713e15d307f" media=application/vnd.docker.image.rootfs.diff.tar.gzip size=123555
time="2024-11-04T09:40:54.228790020+08:00" level=debug msg="layer unpacked" duration=11.18742ms layer="sha256:f64c8f418cb03a7c30ffe585d63beddccadbae2468a802f6c59ad713e15d307f"
time="2024-11-04T09:40:54.230122120+08:00" level=debug msg="image unpacked" chainID="sha256:da5de361912d14caf254878f4dfc0ac2e97c87b83ef452fa591aa2c99155213a" config="sha256:2324be6c5743ad72bedb72bec9beaf989bd5bace9ed49992566ee2bcda03dcdd" duration=1.28756063s
time="2024-11-04T09:40:54.230158721+08:00" level=debug msg="create image" name="swr.cn-east-3.myhuaweicloud.com/hmirror/rabbitmqoperator/cluster-operator:2.7.0" target="sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c"
time="2024-11-04T09:40:54.231445140+08:00" level=debug msg="Received containerd event timestamp - 2024-11-04 01:40:54.231343703 +0000 UTC, namespace - \"k8s.io\", topic - \"/images/create\""
time="2024-11-04T09:40:54.231525688+08:00" level=info msg="ImageCreate event name:\"swr.cn-east-3.myhuaweicloud.com/hmirror/rabbitmqoperator/cluster-operator:2.7.0\" labels:{key:\"io.cri-containerd.image\" value:\"managed\"}"
time="2024-11-04T09:40:54.232429041+08:00" level=info msg="stop pulling image swr.cn-east-3.myhuaweicloud.com/hmirror/rabbitmqoperator/cluster-operator:2.7.0: active requests=0, bytes read=25973122"
time="2024-11-04T09:40:54.232490307+08:00" level=debug msg="create image" name="sha256:2324be6c5743ad72bedb72bec9beaf989bd5bace9ed49992566ee2bcda03dcdd" target="sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c"
time="2024-11-04T09:40:54.233877091+08:00" level=debug msg="Received containerd event timestamp - 2024-11-04 01:40:54.233710466 +0000 UTC, namespace - \"k8s.io\", topic - \"/images/create\""
time="2024-11-04T09:40:54.233931868+08:00" level=info msg="ImageCreate event name:\"sha256:2324be6c5743ad72bedb72bec9beaf989bd5bace9ed49992566ee2bcda03dcdd\" labels:{key:\"io.cri-containerd.image\" value:\"managed\"}"
time="2024-11-04T09:40:54.234142279+08:00" level=debug msg="create image" name="swr.cn-east-3.myhuaweicloud.com/hmirror/rabbitmqoperator/cluster-operator:2.7.0" target="sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c"
time="2024-11-04T09:40:54.234602279+08:00" level=debug msg="create image" name="swr.cn-east-3.myhuaweicloud.com/hmirror/rabbitmqoperator/cluster-operator@sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c" target="sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c"
time="2024-11-04T09:40:54.236086579+08:00" level=debug msg="Received containerd event timestamp - 2024-11-04 01:40:54.235934993 +0000 UTC, namespace - \"k8s.io\", topic - \"/images/create\""
time="2024-11-04T09:40:54.236127103+08:00" level=info msg="ImageCreate event name:\"swr.cn-east-3.myhuaweicloud.com/hmirror/rabbitmqoperator/cluster-operator@sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c\" labels:{key:\"io.cri-containerd.image\" value:\"managed\"}"
time="2024-11-04T09:40:54.236487865+08:00" level=info msg="Pulled image \"swr.cn-east-3.myhuaweicloud.com/hmirror/rabbitmqoperator/cluster-operator:2.7.0\" with image id \"sha256:2324be6c5743ad72bedb72bec9beaf989bd5bace9ed49992566ee2bcda03dcdd\", repo tag \"swr.cn-east-3.myhuaweicloud.com/hmirror/rabbitmqoperator/cluster-operator:2.7.0\", repo digest \"swr.cn-east-3.myhuaweicloud.com/hmirror/rabbitmqoperator/cluster-operator@sha256:7628e022b4159342a8c4056228023e7030875a131675cae73248dea86875d68c\", size \"25963651\" in 2.310026116s"
time="2024-11-04T09:40:54.236517667+08:00" level=info msg="PullImage \"swr.cn-east-3.myhuaweicloud.com/hmirror/rabbitmqoperator/cluster-operator:2.7.0\" returns image reference \"sha256:2324be6c5743ad72bedb72bec9beaf989bd5bace9ed49992566ee2bcda03dcdd\""
time="2024-11-04T09:23:59.283997031+08:00" level=debug msg="do request" host=swr.cn-east-3.myhuaweicloud.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.7.15-k3s1 request.method=HEAD url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/2.7.0?ns=docker.io"time="2024-11-04T09:23:59.324497576+08:00" level=debug msg="fetch response received" host=swr.cn-east-3.myhuaweicloud.com response.header.connection=keep-alive response.header.content-length=61 response.header.content-type="application/json;charset=UTF-8" response.header.date="Mon, 04 Nov 2024 01:23:59 GMT" response.header.forserver=swr response.header.keep-alive="timeout=300" response.header.server="Web Server" response.header.www-authenticate="Bearer realm=\"https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/\",service=\"dockyard\",scope=\"repository:hmirror/rabbitmqoperator/cluster-operator:\"" response.status="401 Unauthorized" url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/2.7.0?ns=docker.io"time="2024-11-04T09:23:59.324542691+08:00" level=debug msg=Unauthorized header="Bearer realm=\"https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/\",service=\"dockyard\",scope=\"repository:hmirror/rabbitmqoperator/cluster-operator:\"" host=swr.cn-east-3.myhuaweicloud.comtime="2024-11-04T09:23:59.324578114+08:00" level=debug msg="do request" host=swr.cn-east-3.myhuaweicloud.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.7.15-k3s1 request.method=HEAD url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/2.7.0?ns=docker.io"time="2024-11-04T09:23:59.617911572+08:00" level=info msg="trying next host" error="failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/?scope=repository%3Ahmirror%2Frabbitmqoperator%2Fcluster-operator%3A&scope=repository%3Arabbitmqoperator%2Fcluster-operator%3Apull&service=dockyard: 404 Not Found" host=swr.cn-east-3.myhuaweicloud.com
msg=Unauthorized header="Bearer realm=\"https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/\",service=\"dockyard\",scope=\"repository:hmirror/rabbitmqoperator/cluster-operator:\"" host=swr.cn-east-3.myhuaweicloud.comunexpected status from GET request to https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/?scope=repository%3Ahmirror%2Frabbitmqoperator%2Fcluster-operator%3A&scope=repository%3Arabbitmqoperator%2Fcluster-operator%3Apull&service=dockyard: 404 Not Found"
It looks like containerd is failing to modify the repository name in the second scope. I'm not sure why that'd be, but I do note that you're using containerd v1.7.15-k3s1 which we haven't shipped since May of 2024. Please try a newer release of K3s, and let us know if the issue persists.
codering
commented
3 weeks ago @brandond same error in newer version (v1.29.10+k3s1)
crictl pull docker.io/rabbitmqoperator/cluster-operator:2.7.0
time="2024-11-05T09:56:23.718495322+08:00" level=info msg="PullImage \"docker.io/rabbitmqoperator/cluster-operator:2.7.0\""
time="2024-11-05T09:56:23.718550765+08:00" level=debug msg="PullImage \"docker.io/rabbitmqoperator/cluster-operator:2.7.0\" with snapshotter overlayfs"
time="2024-11-05T09:56:23.721729505+08:00" level=debug msg="loading host directory" dir=/var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io
time="2024-11-05T09:56:23.721921417+08:00" level=debug msg=resolving host=swr.cn-east-3.myhuaweicloud.com
time="2024-11-05T09:56:23.721945017+08:00" level=debug msg="do request" host=swr.cn-east-3.myhuaweicloud.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.7.22-k3s1 request.method=HEAD url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/2.7.0?ns=docker.io"
time="2024-11-05T09:56:23.754841177+08:00" level=debug msg="fetch response received" host=swr.cn-east-3.myhuaweicloud.com response.header.connection=keep-alive response.header.content-length=61 response.header.content-type="application/json;charset=UTF-8" response.header.date="Tue, 05 Nov 2024 01:56:24 GMT" response.header.forserver=swr response.header.keep-alive="timeout=300" response.header.server="Web Server" response.header.www-authenticate="Bearer realm=\"https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/\",service=\"dockyard\",scope=\"repository:hmirror/rabbitmqoperator/cluster-operator:\"" response.status="401 Unauthorized" url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/2.7.0?ns=docker.io"
time="2024-11-05T09:56:23.754880209+08:00" level=debug msg=Unauthorized header="Bearer realm=\"https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/\",service=\"dockyard\",scope=\"repository:hmirror/rabbitmqoperator/cluster-operator:\"" host=swr.cn-east-3.myhuaweicloud.com
time="2024-11-05T09:56:23.754922509+08:00" level=debug msg="do request" host=swr.cn-east-3.myhuaweicloud.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.7.22-k3s1 request.method=HEAD url="https://swr.cn-east-3.myhuaweicloud.com/v2/hmirror/rabbitmqoperator/cluster-operator/manifests/2.7.0?ns=docker.io"
time="2024-11-05T09:56:24.024202297+08:00" level=info msg="trying next host" error="failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/?scope=repository%3Ahmirror%2Frabbitmqoperator%2Fcluster-operator%3A&scope=repository%3Arabbitmqoperator%2Fcluster-operator%3Apull&service=dockyard: 404 Not Found" host=swr.cn-east-3.myhuaweicloud.com
time="2024-11-05T09:56:24.024248431+08:00" level=debug msg=resolving host=registry-1.docker.io
time="2024-11-05T09:56:24.024269825+08:00" level=debug msg="do request" host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.7.22-k3s1 request.method=HEAD url="https://registry-1.docker.io/v2/rabbitmqoperator/cluster-operator/manifests/2.7.0"
time="2024-11-05T09:56:24.262991073+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://registry-1.docker.io/v2/rabbitmqoperator/cluster-operator/manifests/2.7.0\": read tcp 192.168.0.101:47928->54.198.86.24:443: read: connection reset by peer" host=registry-1.docker.io
time="2024-11-05T09:56:24.264423988+08:00" level=error msg="PullImage \"docker.io/rabbitmqoperator/cluster-operator:2.7.0\" failed" error="failed to pull and unpack image \"docker.io/rabbitmqoperator/cluster-operator:2.7.0\": failed to resolve reference \"docker.io/rabbitmqoperator/cluster-operator:2.7.0\": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/?scope=repository%3Ahmirror%2Frabbitmqoperator%2Fcluster-operator%3A&scope=repository%3Arabbitmqoperator%2Fcluster-operator%3Apull&service=dockyard: 404 Not Found"
time="2024-11-05T09:56:24.264497137+08:00" level=info msg="stop pulling image docker.io/rabbitmqoperator/cluster-operator:2.7.0: active requests=0, bytes read=194"
I'm not sure why you have two scopes there, but I can see if I can reproduce this internally and figure out if this is a containerd bug, or what.
brandond
commented
3 weeks ago The issue here is that there are two scopes in the auth request. One of them comes from the Unauthorized response:
Bearer realm=\"https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/\",service=\"dockyard\",scope=\"repository:hmirror/rabbitmqoperator/cluster-operator:\"
The other is generated internally by containerd based on the repository of the image being pulled, but unfortunately this is not properly rewritten. I suspect that most auth providers just ignore the unknown claim for the unmodified registry scope and use the one that they do recognize, or perhaps only evaluate the first scope, but this one apparently returns a 404 for the bearer token request because it does not recognize all of the requested scopes. To be fair, this is probably safer behavior.
https://swr.cn-east-3.myhuaweicloud.com/swr/auth/v2/registry/auth/?scope=repository%3Ahmirror%2Frabbitmqoperator%2Fcluster-operator%3A&scope=repository%3Arabbitmqoperator%2Fcluster-operator%3Apull&service=dockyard: 404 Not Foundscope=repository:hmirror/rabbitmqoperator/cluster-operator: scope=repository:rabbitmqoperator/cluster-operator:pull
I believe I have addressed this in https://github.com/brandond/containerd/commit/c18a4212c76eed6dc52d94c5f48fc6598e140712
Note for QA: This may be difficult to reproduce, as it requires specific behavior from the registry auth provider. It appears that only artifactory and huawei cloud are affected?
Environmental Info: K3s Version: k3s version v1.30.5+k3s1 go version go1.22.6
Node(s) CPU architecture, OS, and Version: Linux 5.14.0-284.30.1.el9_2.x86_64 https://github.com/k3s-io/k3s/issues/1 SMP PREEMPT_DYNAMIC Fri Aug 25 09:13:12 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
Cluster Configuration: 1 server, 1 agent
Describe the bug: I set up two mirror registries with rewrite configuration in registries.yaml in k3s(both server and agent)
The first mirror registry configuration works well. I can start a pod that needs to pull image from
icr.io/cpopen/edge-operator-catalog@sha256:4f9725b23c8560eae25be0a9fac01c74c9d4a9fee8200e31aad9842f7c338433, but actually pull image from mirror registry:https://docker-na-public.artifactory.test.com/se-next-gen-docker-local/edge-operator-catalog@sha256:4f9725b23c8560eae25be0a9fac01c74c9d4a9fee8200e31aad9842f7c338433successfullyHowever, the second mirror registry configuration does not work properly. Another pod which needs to pull image from cp.icr.io/cp/se-data-center-edge/mini-test@sha256:c718d3f996061aef92966a2171713af1cfdbac93cbea7a753107e3d5430c3687 can not pull image from mirror registry https://docker-na-public.artifactory.test.com/se-next-gen-docker-local/mini-test@sha256:c718d3f996061aef92966a2171713af1cfdbac93cbea7a753107e3d5430c3687.
The error shows
The thing is I could manually pull that image from mirror registry
Is that rewrite configuration wrong for the second mirror registry?
Steps To Reproduce:
Expected behavior:
Actual behavior:
Additional context / logs: