Cannot delete manually loaded docker images(`kind load docker-image`) inside control-plane container

[vinayakbhat24]

What happened: We cannot delete images loaded to Kind cluster using the command kind load docker-image .... The issue is that it increases the RAM usage every time new images are loaded into the cluster since we cannot delete old images.

What you expected to happen: Inside the docker container control plane crictl rmi IMAGE_ID -> should delete a respective image, but not deleting. We also tried with a non-truncated IMAGE ID, but no success.

How to reproduce it (as minimally and precisely as possible):

• Load a docker image to kind cluster using the command kind load docker-image
• Try to delete the loaded docker image - crictl rmi IMAGE_ID

Anything else we need to know?: We tried to debug the issue using debug flags everywhere. The issue is found to be empty RepoTags for loaded images. This can be easily seen with the command crictl inspecti {ImageID}

Also, we tried using ctr command, but no success.

Environment:

• kind version: 0.17.0:
• Runtime info (docker info):

  Client:
  Context:    default
  Debug Mode: false
  Plugins:
  buildx: Docker Buildx (Docker Inc.)
  Version:  v0.10.2
  Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
  Version:  v2.16.0
  Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
  Version:  v0.23.0
  Path:     /usr/libexec/docker/cli-plugins/docker-scan
  Server:
  Server Version: 23.0.1

• OS :

  Description:    Ubuntu 20.04.5 LTS
  Release:    20.04
  Codename:   focal

• Kubernetes version

  Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d", GitTreeState:"clean", BuildDate:"2022-12-08T19:58:30Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
  Kustomize Version: v4.5.7

### Tasks

[BenTheElder]

You have to also delete the digest-only reference created by image importing.

As long as there is another reference to the image you're just deleting the tag.

/kind support /remove-kind bug

[BenTheElder]

aside: it should be disk usage, not RAM, unless somehow your docker volumes are memory backed

[vinayakbhat24]

@BenTheElder Tried out the following commands to delete digest reference, but no luck!

#crictl images --digests=true --no-trunc
docker.io/library/import-2023-09-26                       <none>               sha256:cf5db0e942a3bb33a6394331f7216ed7eceb94d0bfa117b628dc40d799150796   sha256:29504ba812afe36081ccbdf84f813675998aed17b66e36985035ded7f4296b82   128MB

#crictl rmi docker.io/library/import-2023-09-26@sha256:cf5db0e942a3bb33a6394331f7216ed7eceb94d0bfa117b628dc40d799150796
ERRO[0000] no such image docker.io/library/import-2023-09-26@sha256:cf5db0e942a3bb33a6394331f7216ed7eceb94d0bfa117b628dc40d799150796 
FATA[0000] unable to remove the image(s)

[BenTheElder]

I'm not sure if crictl supports this properly, you might have to use ctr -n=k8s.io

[victork0917]

@vinayakbhat24 Did you manage to resolve the issue? If yes, could you please provide the command you use? Thanks.

[vinayakbhat24]

@victork0917 I have not been able to resolve the issue yet.

[rriski]

Related: https://github.com/containerd/containerd/issues/7698

[StLeoX]

This bug can still be reproduced in kind v0.23.0. And I used load image-archive to get around this problem.

[BenTheElder]

Please try v0.24, which is the latest release and ships with a set of images that have newer containerd

[subhankarc]

Hi @BenTheElder

I am facing the same problem on 0.24.0 as well.

# crictl images --digests=true --no-trunc | grep nginx
docker.io/library/nginx                         1.16.0               sha256:66f3d00a86a04a2bc78b7682a0fb9b8dff86fe56c4662245253a39707b00579c   sha256:162eee5cdbffd3ef183cc6d8b7a16c3aa7f5516dbbeab986204a1b1077561974   107MB
root@kind-control-plane:/# 
root@kind-control-plane:/# 
root@kind-control-plane:/# crictl rmi docker.io/library/nginx@sha256:66f3d00a86a04a2bc78b7682a0fb9b8dff86fe56c4662245253a39707b00579c
ERRO[0000] no such image docker.io/library/nginx@sha256:66f3d00a86a04a2bc78b7682a0fb9b8dff86fe56c4662245253a39707b00579c 
FATA[0000] unable to remove the image(s)

$ kind version
kind v0.24.0 go1.23.1 linux/arm64

[BenTheElder]

Was this image ever run in the cluster?

Have you tried using the tag instead of the digest?

[stmcginnis]

Either of these should work:

# crictl rmi nginx:1.16.0                                                           
Deleted: docker.io/library/nginx:1.16.0

Or

root@kind-control-plane:/# crictl images | grep nginx             
docker.io/library/nginx                         latest               39286ab8a5e14       71MB
root@kind-control-plane:/# crictl rmi 39286ab8a5e14
Deleted: docker.io/library/nginx:latest

[subhankarc]

@BenTheElder @stmcginnis

Thanks! yes only the tag works.

also ctr -n=k8s.io images rm docker.io/library/nginx:1.16.0 works.

[BenTheElder]

This might be a bug in crictl? docker.io/library defaulting normalization (at least from the point of view of compatibility with kubernetes pods) might be missing somewhere.