crictl pulls incorrect image with nightly or latest tag

[bjiang27]

What happened:

when using crictl to pull image with nightly or latest tag, an older version of the image is pulled.

What you expected to happen:

when using crictl to pull image with nightly or latest tag, the image with the nightly or latest tag respectively should be pulled.

Anything else we need to know?:

when using crictl to pull image with semantic version tag, the correct image is pulled.

[SergeyKanzhelev]

more info is needed to troubleshoot:

• container runtime
• verbose logs from crictl
• as verbose logs as you can give from the container runtime

Typical causes: mirror configured for your registry that is slow to update the tag. Registry failed to respond or timed out and older image was used. There may be more - this is why logs would be helpful to troubleshoot.

[bjiang27]

Container Runtime Version: containerd://1.6.32

verbose logs from crictl:

DEBU[0000] get image connection                         
DEBU[0000] ListImagesRequest: &ListImagesRequest{Filter:&ImageFilter{Image:&ImageSpec{Image:,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},},} 
DEBU[0000] ListImagesResponse: &ListImagesResponse{Images:[]*Image{&Image{Id:sha256:fb491fb44f20073afcbea4d34ccd66f0b0ed754b010a671554975ea235c5d478,RepoTags:[docker.io/dellemc/dell-csm-operator:v1.5.1],RepoDigests:[docker.io/dellemc/dell-csm-operator@sha256:31de1529604bef1e9f1391585126317a93d3f997dd3a52b93b9dae7ab562bcab],Size_:34648173,Uid:&Int64Value{Value:1001,},Username:,Spec:nil,Pinned:false,},&Image{Id:sha256:78047822fe2eef446960383261b0a579215b32bc5587482b4234585453e614e8,RepoTags:[docker.io/dellemc/dell-csm-operator:latest],RepoDigests:[docker.io/dellemc/dell-csm-operator@sha256:d222214b990c992e7d5922d7b136ae1babc5e54469658082c1e514a75e16fa70],Size_:33361333,Uid:&Int64Value{Value:1001,},Username:,Spec:nil,Pinned:false,},&Image{Id:sha256:8d4fb504c29ed917231292387fb8095f58169971238ff2d76ff50e0f07e40834,RepoTags:[docker.io/dellemc/dell-csm-operator:nightly],RepoDigests:[docker.io/dellemc/dell-csm-operator@sha256:e301e28b1d9243746edc86dddd4ed3202bbb009366d2e97466602b2887a47dff],Size_:33062097,Uid:&Int64Value{Value:1001,},Username:,Spec:nil,Pinned:false,},},} 
IMAGE                                 TAG                 IMAGE ID            SIZE
docker.io/dellemc/dell-csm-operator   latest              78047822fe2ee       33.4MB
docker.io/dellemc/dell-csm-operator   nightly             8d4fb504c29ed       33.1MB
docker.io/dellemc/dell-csm-operator   v1.5.1              fb491fb44f200       34.6MB

the latest version should be v1.5.1, however the pulled version is v1.4.2

snippets of crictl inspecti: "license": "Dell CSM Operator Apache License", "maintainer": "Red Hat, Inc.", "name": "dell-csm-operator", "release": "13", "summary": "Operator for installing Dell CSI Drivers and Dell CSM Modules", "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi9/ubi-micro/images/9.3-13", "vcs-ref": "303dc144996db01765d69e8ea45d2d617d953e42", "vcs-type": "git", "vendor": "Dell Inc.", "version": "1.4.2"

image that has the issue: https://hub.docker.com/r/dellemc/dell-csm-operator/tags

[bjiang27]

we are not seeing this error when using docker pull. we are only seeing this error when using crictl.

[kwilczynski]

I am unable to reproduce this problem when using CRI-O, per:

[root@kwilczynski-dev2-xg5f8-worker-a-q9xhm ~]# crictl -D pull dellemc/dell-csm-operator:v1.5.1
DEBU[0000] get image connection                         
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:dellemc/dell-csm-operator:v1.5.1,Annotations:map[string]string{},UserSpecifiedImage:,},Auth:nil,SandboxConfig:nil,} 
DEBU[0006] PullImageResponse: &PullImageResponse{ImageRef:docker.io/dellemc/dell-csm-operator@sha256:31de1529604bef1e9f1391585126317a93d3f997dd3a52b93b9dae7ab562bcab,} 
Image is up to date for docker.io/dellemc/dell-csm-operator@sha256:31de1529604bef1e9f1391585126317a93d3f997dd3a52b93b9dae7ab562bcab

[root@kwilczynski-dev2-xg5f8-worker-a-q9xhm ~]# crictl image | grep dell
docker.io/dellemc/dell-csm-operator              latest              fb491fb44f200       86.5MB
docker.io/dellemc/dell-csm-operator              v1.5.1              fb491fb44f200       86.5MB
docker.io/dellemc/dell-csm-operator              nightly             5360b740f9113       142MB

[root@kwilczynski-dev2-xg5f8-worker-a-q9xhm ~]# crictl -D pull dellemc/dell-csm-operator:latest
DEBU[0000] get image connection                         
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:dellemc/dell-csm-operator:latest,Annotations:map[string]string{},UserSpecifiedImage:,},Auth:nil,SandboxConfig:nil,} 
DEBU[0005] PullImageResponse: &PullImageResponse{ImageRef:docker.io/dellemc/dell-csm-operator@sha256:31de1529604bef1e9f1391585126317a93d3f997dd3a52b93b9dae7ab562bcab,} 
Image is up to date for docker.io/dellemc/dell-csm-operator@sha256:31de1529604bef1e9f1391585126317a93d3f997dd3a52b93b9dae7ab562bcab

[root@kwilczynski-dev2-xg5f8-worker-a-q9xhm ~]# crictl inspecti docker.io/dellemc/dell-csm-operator:latest | grep -E '"version"'
      "version": "1.5.1",
          "version": "1.5.1",

[root@kwilczynski-dev2-xg5f8-worker-a-q9xhm ~]# crictl inspecti docker.io/dellemc/dell-csm-operator:latest | jq '.status.repoTags'
[
  "docker.io/dellemc/dell-csm-operator:latest",
  "docker.io/dellemc/dell-csm-operator:v1.5.1"
]

[root@kwilczynski-dev2-xg5f8-worker-a-q9xhm ~]# crictl inspecti docker.io/dellemc/dell-csm-operator:nightly | grep -E '"version"'
      "version": "1.6.0",
          "version": "1.6.0",

[root@kwilczynski-dev2-xg5f8-worker-a-q9xhm ~]# crictl inspecti docker.io/dellemc/dell-csm-operator:nightly | jq '.status.repoTags'
[
  "docker.io/dellemc/dell-csm-operator:nightly"
]

This was done using the following:

• CRI-O: crio version 1.28.7-4.rhaos4.15.git745ee5b.el9
• crictl: crictl version 1.28.0

[SergeyKanzhelev]

@bjiang27 do you know if any mirrors are configured for dockerhub in containerd which are not configured in docker? The mirror slagging behind is a common reason of problems like this. Is this issue resolves itself after some time or the tag being stuck to the specific label (i.e. it is lagging behind in the morning and resolves itself by evening)?

[bjiang27]

the issue is with nightly or latest tag. semantic version tag is working.

[kwilczynski]

@bjiang27, I update my comment with a pull against latest, with pulled the same image (SHA) as the v1.5.1 tag.

[SergeyKanzhelev]

the issue is with nightly or latest tag. semantic version tag is working.

exactly, mirrors may not detect the change of a tag as quickly as an appearance of a new version. Please check config for the mirror and try to get continerd logs to understand what does it actually pull

[SergeyKanzhelev]

also you may try to use ctr with verbose. It will emulate some of the logic contianerd does. But if the mirror is configured for CRI, ctr will not use it and will likely work the same way as docker.

[bjiang27]

I do not think that we have any mirrors configured.

[SergeyKanzhelev]

@bjiang27 there are still some questions unanswered. We can only help troubleshoot with extra information. Specifically, check for mirror config (may be in: /etc/containerd/config.toml), containerd logs, and try to use ctr with http trace and see whether it will give the right image.

[bjiang27]

thank you @SergeyKanzhelev @kwilczynski we could identify the mirrors due to the above comment, and we have removed the mirrors from the /etc/containerd/config.toml and the pull was successful with respective image version.