Load muliple local images to kind cluster can corrupt

tthvo commented 1 year ago

What happened:

When trying to load multiple images with kind load docker-image can result in incorrect state.

In my case, I tried to load 2 images with different tags. The command succeeded. However, upon inspecting the node, only one image (i.e. one image entry) is loaded and 2 different tags are set to point to it.

What you expected to happen:

Two images should be loaded correctly with 2 separate entries in node status.

How to reproduce it (as minimally and precisely as possible):

With 2 images built locally:

REPOSITORY                         TAG                 IMAGE ID      CREATED            SIZE
quay.io/cryostat/cryostat3-db      dev                 89d4319a4478  About an hour ago  414 MB
quay.io/cryostat/cryostat3         dev                 8ed0252bb0cc  About an hour ago  485 MB

Launch a kind cluster

$ kind create cluster

Load the images into cluster nodes

$ kind load docker-image quay.io/cryostat/cryostat3:dev quay.io/cryostat/cryostat3-db:dev
enabling experimental podman provider
Image: "quay.io/cryostat/cryostat3:dev" with ID "8ed0252bb0cc12fa3fe3ca2d3de758980ae4302224bb1e6907c57f11cd2abf20" not yet present on node "kind-control-plane", loading...
Image: "quay.io/cryostat/cryostat3-db:dev" with ID "89d4319a44782dd57383aa583db3ad535a1b4d416a7e020e183acec1ddc28493" not yet present on node "kind-control-plane", loading...

Notice a single image is loaded but pointed to by 2 tags specified above

$ oc get node/kind-control-plane -o yaml | yq .status.images
- names:
    - docker.io/library/import-2023-07-13@sha256:07369612660e99adc2bf45888d1d0233f111b19a0d02f3985457798f3dc4242e
    - quay.io/cryostat/cryostat3-db:dev
    - quay.io/cryostat/cryostat3:dev
  sizeBytes: 484978064
- names:
    - registry.k8s.io/etcd:3.5.7-0
  sizeBytes: 101639218
- names:
    - docker.io/library/import-2023-05-12@sha256:e47e5ddd7105c32de2fee0049d89ebb3253262694f678b92c70678f4e7b1d1c1
    - registry.k8s.io/kube-apiserver:v1.27.1
  sizeBytes: 83436031
- names:
    - docker.io/library/import-2023-05-12@sha256:c2a0c5ea531ff4e7f4c4e44fe5969bbe8e77dbdb6171961c81e07454648c6526
    - registry.k8s.io/kube-controller-manager:v1.27.1
  sizeBytes: 74407565
- names:
    - docker.io/library/import-2023-05-12@sha256:a9d9b674a55b137b1f10bbcd7f0e43b7568d20c8df5bf969d1bb8a37f3d365db
    - registry.k8s.io/kube-proxy:v1.27.1
  sizeBytes: 72702973
- names:
    - docker.io/library/import-2023-05-12@sha256:8b1ab2b02004fc7ae90a8c562dfdf13d63361770d15f426feb8f79779557277f
    - registry.k8s.io/kube-scheduler:v1.27.1
  sizeBytes: 59781261
- names:
    - docker.io/kindest/kindnetd:v20230511-dc714da8
  sizeBytes: 27731571
- names:
    - docker.io/kindest/local-path-provisioner:v20230511-dc714da8
  sizeBytes: 19351145
- names:
    - registry.k8s.io/coredns/coredns:v1.10.1
  sizeBytes: 16190758
- names:
    - docker.io/kindest/local-path-helper:v20230510-486859a6
  sizeBytes: 3052318
- names:
    - registry.k8s.io/pause:3.7
  sizeBytes: 311278

Anything else we need to know?:

If I load one image at time, everything is working as expected (i.e. 2 separate entries for 2 images with correct sum).

Environment:

kind version: (use kind version): kind v0.19.0 go1.20.3 linux/amd64 with rootless podman.
Runtime info: (use docker info or podman info): podman v4.5.1 linux/amd64
OS (e.g. from /etc/os-release): Fedora 38 (Workstate Edition)
Kubernetes version: (use kubectl version): v1.24.1
Any proxies or other special environment settings?:

BenTheElder commented 1 year ago

$ oc get node/kind-control-plane -o yaml | yq .status.images

FWIW this isn't the best way to get image listings, since it relies on node status which is an incomplete and potential stale list of images. docker exec $node crictl images.

kind load docker-image [...] podman v4.5.1 linux/amd64

This command doesn't support podman (see other tracking issues about replacing it), podman save foo/bar:baz | kind load image-archive - is supported however.

Still sounds like a bug, can you create a minimal reproducer that we can actually reproduce? (since we don't have the sources for this particular image)

tthvo commented 1 year ago

This command doesn't support podman (see other tracking issues about replacing it), podman save foo/bar:baz | kind load image-archive - is supported however.

Thanks @BenTheElder for the suggestion. I didn't realize I forgot to mention that we are using podman-docker package to emulate docker CLI with podman.

I guess the root issue is that podman is not supported for this command? Tho, I thought it would fail early as shown in other issue, for example, https://github.com/kubernetes-sigs/kind/issues/2417? And it seemed to work when loading a single image at a time.

Still sounds like a bug, can you create a minimal reproducer that we can actually reproduce? (since we don't have the sources for this particular image)

Can u try? Notice how the 2 images have the same ID.

$ kind create cluster
$ kind load docker-image quay.io/cryostat/cryostat:latest quay.io/andrewazores/vertx-fib-demo:0.13.0
$ docker exec kind-control-plane crictl images
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
IMAGE                                      TAG                  IMAGE ID            SIZE
docker.io/kindest/kindnetd                 v20230511-dc714da8   b0b1fa0f58c6e       27.7MB
docker.io/kindest/local-path-helper        v20230510-486859a6   be300acfc8622       3.05MB
docker.io/kindest/local-path-provisioner   v20230511-dc714da8   ce18e076e9d4b       19.4MB
quay.io/andrewazores/vertx-fib-demo        0.13.0               0b93452b28ab6       492MB
quay.io/cryostat/cryostat                  latest               0b93452b28ab6       492MB
registry.k8s.io/coredns/coredns            v1.10.1              ead0a4a53df89       16.2MB
registry.k8s.io/etcd                       3.5.7-0              86b6af7dd652c       102MB
registry.k8s.io/kube-apiserver             v1.27.1              b9461467a89fc       83.4MB
registry.k8s.io/kube-controller-manager    v1.27.1              cee852c75b211       74.4MB
registry.k8s.io/kube-proxy                 v1.27.1              431553d4c70b5       72.7MB
registry.k8s.io/kube-scheduler             v1.27.1              731b57d3459ce       59.8MB
registry.k8s.io/pause                      3.7                  221177c6082a8       311kB

BenTheElder commented 11 months ago

Can u try? Notice how the 2 images have the same ID.

Thanks, it may be a bit as I don't have ready access to a machine with podman (it's difficult to install both podman and docker on many distros due to runc conflicts and I need docker for other purposes) and I'm doing a lot at the moment.

I guess the root issue is that podman is not supported for this command? Tho, I thought it would fail early as shown in other issue, for example, https://github.com/kubernetes-sigs/kind/issues/2417? And it seemed to work when loading a single image at a time.

For some of the other codepaths we do detect actually docker vs podman but we weren't expecting people to use the docker subcommand(s) with podman and podman to be incompatible.

There's an open tracking issue about creating an alternate command that doesn't break the behavior of the existing docker command but introduces a new command that intelligently selects between image sources including podman but there hasn't been much work yet #2038.

kubernetes-sigs / kind

Load muliple local images to kind cluster can corrupt #3306