Closed bjiang27 closed 2 months ago
more info is needed to troubleshoot:
Typical causes: mirror configured for your registry that is slow to update the tag. Registry failed to respond or timed out and older image was used. There may be more - this is why logs would be helpful to troubleshoot.
Container Runtime Version: containerd://1.6.32
verbose logs from crictl:
DEBU[0000] get image connection
DEBU[0000] ListImagesRequest: &ListImagesRequest{Filter:&ImageFilter{Image:&ImageSpec{Image:,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},},}
DEBU[0000] ListImagesResponse: &ListImagesResponse{Images:[]*Image{&Image{Id:sha256:fb491fb44f20073afcbea4d34ccd66f0b0ed754b010a671554975ea235c5d478,RepoTags:[docker.io/dellemc/dell-csm-operator:v1.5.1],RepoDigests:[docker.io/dellemc/dell-csm-operator@sha256:31de1529604bef1e9f1391585126317a93d3f997dd3a52b93b9dae7ab562bcab],Size_:34648173,Uid:&Int64Value{Value:1001,},Username:,Spec:nil,Pinned:false,},&Image{Id:sha256:78047822fe2eef446960383261b0a579215b32bc5587482b4234585453e614e8,RepoTags:[docker.io/dellemc/dell-csm-operator:latest],RepoDigests:[docker.io/dellemc/dell-csm-operator@sha256:d222214b990c992e7d5922d7b136ae1babc5e54469658082c1e514a75e16fa70],Size_:33361333,Uid:&Int64Value{Value:1001,},Username:,Spec:nil,Pinned:false,},&Image{Id:sha256:8d4fb504c29ed917231292387fb8095f58169971238ff2d76ff50e0f07e40834,RepoTags:[docker.io/dellemc/dell-csm-operator:nightly],RepoDigests:[docker.io/dellemc/dell-csm-operator@sha256:e301e28b1d9243746edc86dddd4ed3202bbb009366d2e97466602b2887a47dff],Size_:33062097,Uid:&Int64Value{Value:1001,},Username:,Spec:nil,Pinned:false,},},}
IMAGE TAG IMAGE ID SIZE
docker.io/dellemc/dell-csm-operator latest 78047822fe2ee 33.4MB
docker.io/dellemc/dell-csm-operator nightly 8d4fb504c29ed 33.1MB
docker.io/dellemc/dell-csm-operator v1.5.1 fb491fb44f200 34.6MB
the latest version should be v1.5.1, however the pulled version is v1.4.2
snippets of crictl inspecti: "license": "Dell CSM Operator Apache License", "maintainer": "Red Hat, Inc.", "name": "dell-csm-operator", "release": "13", "summary": "Operator for installing Dell CSI Drivers and Dell CSM Modules", "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi9/ubi-micro/images/9.3-13", "vcs-ref": "303dc144996db01765d69e8ea45d2d617d953e42", "vcs-type": "git", "vendor": "Dell Inc.", "version": "1.4.2"
image that has the issue: https://hub.docker.com/r/dellemc/dell-csm-operator/tags
we are not seeing this error when using docker pull. we are only seeing this error when using crictl.
I am unable to reproduce this problem when using CRI-O, per:
[root@kwilczynski-dev2-xg5f8-worker-a-q9xhm ~]# crictl -D pull dellemc/dell-csm-operator:v1.5.1
DEBU[0000] get image connection
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:dellemc/dell-csm-operator:v1.5.1,Annotations:map[string]string{},UserSpecifiedImage:,},Auth:nil,SandboxConfig:nil,}
DEBU[0006] PullImageResponse: &PullImageResponse{ImageRef:docker.io/dellemc/dell-csm-operator@sha256:31de1529604bef1e9f1391585126317a93d3f997dd3a52b93b9dae7ab562bcab,}
Image is up to date for docker.io/dellemc/dell-csm-operator@sha256:31de1529604bef1e9f1391585126317a93d3f997dd3a52b93b9dae7ab562bcab
[root@kwilczynski-dev2-xg5f8-worker-a-q9xhm ~]# crictl image | grep dell
docker.io/dellemc/dell-csm-operator latest fb491fb44f200 86.5MB
docker.io/dellemc/dell-csm-operator v1.5.1 fb491fb44f200 86.5MB
docker.io/dellemc/dell-csm-operator nightly 5360b740f9113 142MB
[root@kwilczynski-dev2-xg5f8-worker-a-q9xhm ~]# crictl -D pull dellemc/dell-csm-operator:latest
DEBU[0000] get image connection
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:dellemc/dell-csm-operator:latest,Annotations:map[string]string{},UserSpecifiedImage:,},Auth:nil,SandboxConfig:nil,}
DEBU[0005] PullImageResponse: &PullImageResponse{ImageRef:docker.io/dellemc/dell-csm-operator@sha256:31de1529604bef1e9f1391585126317a93d3f997dd3a52b93b9dae7ab562bcab,}
Image is up to date for docker.io/dellemc/dell-csm-operator@sha256:31de1529604bef1e9f1391585126317a93d3f997dd3a52b93b9dae7ab562bcab
[root@kwilczynski-dev2-xg5f8-worker-a-q9xhm ~]# crictl inspecti docker.io/dellemc/dell-csm-operator:latest | grep -E '"version"'
"version": "1.5.1",
"version": "1.5.1",
[root@kwilczynski-dev2-xg5f8-worker-a-q9xhm ~]# crictl inspecti docker.io/dellemc/dell-csm-operator:latest | jq '.status.repoTags'
[
"docker.io/dellemc/dell-csm-operator:latest",
"docker.io/dellemc/dell-csm-operator:v1.5.1"
]
[root@kwilczynski-dev2-xg5f8-worker-a-q9xhm ~]# crictl inspecti docker.io/dellemc/dell-csm-operator:nightly | grep -E '"version"'
"version": "1.6.0",
"version": "1.6.0",
[root@kwilczynski-dev2-xg5f8-worker-a-q9xhm ~]# crictl inspecti docker.io/dellemc/dell-csm-operator:nightly | jq '.status.repoTags'
[
"docker.io/dellemc/dell-csm-operator:nightly"
]
This was done using the following:
@bjiang27 do you know if any mirrors are configured for dockerhub in containerd which are not configured in docker? The mirror slagging behind is a common reason of problems like this. Is this issue resolves itself after some time or the tag being stuck to the specific label (i.e. it is lagging behind in the morning and resolves itself by evening)?
the issue is with nightly or latest tag. semantic version tag is working.
@bjiang27, I update my comment with a pull against latest, with pulled the same image (SHA) as the v1.5.1 tag.
the issue is with nightly or latest tag. semantic version tag is working.
exactly, mirrors may not detect the change of a tag as quickly as an appearance of a new version. Please check config for the mirror and try to get continerd logs to understand what does it actually pull
also you may try to use ctr
with verbose. It will emulate some of the logic contianerd does. But if the mirror is configured for CRI, ctr
will not use it and will likely work the same way as docker.
I do not think that we have any mirrors configured.
@bjiang27 there are still some questions unanswered. We can only help troubleshoot with extra information. Specifically, check for mirror config (may be in: /etc/containerd/config.toml), containerd logs, and try to use ctr
with http trace and see whether it will give the right image.
thank you @SergeyKanzhelev @kwilczynski we could identify the mirrors due to the above comment, and we have removed the mirrors from the /etc/containerd/config.toml and the pull was successful with respective image version.
What happened:
when using crictl to pull image with nightly or latest tag, an older version of the image is pulled.
What you expected to happen:
when using crictl to pull image with nightly or latest tag, the image with the nightly or latest tag respectively should be pulled.
Anything else we need to know?:
when using crictl to pull image with semantic version tag, the correct image is pulled.