search

k3d-io / k3d

Little helper to run CNCF's k3s in Docker
https://k3d.io/
MIT License
5.35k stars 456 forks source link

[HELP] Importing an image into a second cluster *removes* that image from the previous cluster #468

Open NikTJ777 opened 3 years ago

NikTJ777 commented 3 years ago

What did you do

What did you expect to happen

That the image would be in both clusters simultaneously.

Screenshots or terminal output

$ docker exec -it k3d-ntj-mc-cluster-a-server-0 crictl images IMAGE TAG IMAGE ID SIZE docker.io/kiwigrid/k8s-sidecar 0.1.259 e5722d2b47877 26.4MB docker.io/library/busybox latest b97242f89c8a2 769kB docker.io/rancher/coredns-coredns 1.8.0 296a6d5035e2d 12.9MB docker.io/rancher/klipper-helm v0.3.2 4be09ab862d40 50.7MB docker.io/rancher/klipper-lb v0.1.2 897ce3c5fc8ff 2.71MB docker.io/rancher/library-traefik 1.7.19 aa764f7db3051 24MB docker.io/rancher/local-path-provisioner v0.0.14 e422121c9c5f9 13.4MB docker.io/rancher/metrics-server v0.3.6 9dd718864ce61 10.5MB

$ k3d image import A/B:c.d --cluster ntj-mc-cluster-a INFO[0000] Importing image(s) into cluster 'ntj-mc-cluster-a' INFO[0000] Starting k3d-tools node...
INFO[0001] Starting Node 'k3d-ntj-mc-cluster-a-tools'
INFO[0002] Saving 1 image(s) from runtime...
INFO[0075] Importing images into nodes...
INFO[0075] Importing images from tarball '/k3d/images/k3d-ntj-mc-cluster-a-images-20210128223145.tar' into node 'k3d-ntj-mc-cluster-a-server-0'... INFO[0188] Removing the tarball(s) from image volume... INFO[0189] Removing k3d-tools node...
INFO[0215] Deleted k3d-ntj-mc-cluster-a-tools
INFO[0215] Successfully imported image(s)
INFO[0215] Successfully imported 1 image(s) into 1 cluster(s)

$ docker exec -it k3d-ntj-mc-cluster-a-server-0 crictl images IMAGE TAG IMAGE ID SIZE docker.io/kiwigrid/k8s-sidecar 0.1.259 e5722d2b47877 26.4MB docker.io/library/busybox latest b97242f89c8a2 769kB docker.io/A/B c.d 3d9bc08c19a48 3.45GB docker.io/rancher/coredns-coredns 1.8.0 296a6d5035e2d 12.9MB docker.io/rancher/klipper-helm v0.3.2 4be09ab862d40 50.7MB docker.io/rancher/klipper-lb v0.1.2 897ce3c5fc8ff 2.71MB docker.io/rancher/library-traefik 1.7.19 aa764f7db3051 24MB docker.io/rancher/local-path-provisioner v0.0.14 e422121c9c5f9 13.4MB docker.io/rancher/metrics-server v0.3.6 9dd718864ce61 10.5MB

$ docker exec -it k3d-ntj-mc-cluster-b-server-0 crictl images IMAGE TAG IMAGE ID SIZE docker.io/library/busybox latest b97242f89c8a2 769kB docker.io/rancher/coredns-coredns 1.8.0 296a6d5035e2d 12.9MB docker.io/rancher/klipper-helm v0.3.2 4be09ab862d40 50.7MB docker.io/rancher/klipper-lb v0.1.2 897ce3c5fc8ff 2.71MB docker.io/rancher/library-traefik 1.7.19 aa764f7db3051 24MB docker.io/rancher/local-path-provisioner v0.0.14 e422121c9c5f9 13.4MB docker.io/rancher/metrics-server v0.3.6 9dd718864ce61 10.5MB

$ k3d image import A/B:c.d --cluster ntj-mc-cluster-b INFO[0000] Importing image(s) into cluster 'ntj-mc-cluster-b' INFO[0000] Starting k3d-tools node...
INFO[0001] Starting Node 'k3d-ntj-mc-cluster-b-tools'
INFO[0002] Saving 1 image(s) from runtime...
INFO[0061] Importing images into nodes...
INFO[0061] Importing images from tarball '/k3d/images/k3d-ntj-mc-cluster-b-images-20210128223723.tar' into node 'k3d-ntj-mc-cluster-b-server-0'... INFO[0183] Removing the tarball(s) from image volume... INFO[0184] Removing k3d-tools node...
INFO[0223] Deleted k3d-ntj-mc-cluster-b-tools
INFO[0223] Successfully imported image(s)
INFO[0223] Successfully imported 1 image(s) into 1 cluster(s)

$ docker exec -it k3d-ntj-mc-cluster-b-server-0 crictl images IMAGE TAG IMAGE ID SIZE docker.io/library/busybox latest b97242f89c8a2 769kB docker.io/A/B c.d 3d9bc08c19a48 3.45GB docker.io/rancher/coredns-coredns 1.8.0 296a6d5035e2d 12.9MB docker.io/rancher/klipper-helm v0.3.2 4be09ab862d40 50.7MB docker.io/rancher/klipper-lb v0.1.2 897ce3c5fc8ff 2.71MB docker.io/rancher/library-traefik 1.7.19 aa764f7db3051 24MB docker.io/rancher/local-path-provisioner v0.0.14 e422121c9c5f9 13.4MB docker.io/rancher/metrics-server v0.3.6 9dd718864ce61 10.5MB

$ docker exec -it k3d-ntj-mc-cluster-a-server-0 crictl images IMAGE TAG IMAGE ID SIZE docker.io/kiwigrid/k8s-sidecar 0.1.259 e5722d2b47877 26.4MB docker.io/library/busybox latest b97242f89c8a2 769kB docker.io/rancher/coredns-coredns 1.8.0 296a6d5035e2d 12.9MB docker.io/rancher/klipper-helm v0.3.2 4be09ab862d40 50.7MB docker.io/rancher/klipper-lb v0.1.2 897ce3c5fc8ff 2.71MB docker.io/rancher/library-traefik 1.7.19 aa764f7db3051 24MB docker.io/rancher/local-path-provisioner v0.0.14 e422121c9c5f9 13.4MB docker.io/rancher/metrics-server v0.3.6 9dd718864ce61 10.5MB

Which OS & Architecture

Which version of k3d

Which version of docker

Server: Docker Engine - Community Engine: Version: 19.03.8 API version: 1.40 (minimum version 1.12) Go version: go1.12.17 Git commit: afacb8b Built: Wed Mar 11 01:25:42 2020 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.2.13 GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429 runc: Version: 1.0.0-rc10 GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd docker-init: Version: 0.18.0 GitCommit: fec3683

NikTJ777 commented 3 years ago

Adding some further information to my own question:

It seems that if I wait some minutes more, then the image vanishes from cluster-b as well.

So it seems that k8s is garbage-collecting the images before they have even been used.

If this is so then:

  1. I am hopeful that the eviction.minimum.reclaim setting is to blame - and can be adjusted to stop this;
  2. it would seem that k8s should not attempt to GC an image before it has been used - with perhaps some timeout that is in hours, not minutes or seconds.

I will recreate the cluster without the eviction.minimum.reclaim settings in the hope that fixes it. Otherwise I'll change my workflow, and try creating a pod that uses the image immediately after it has been imported.

iwilltry42 commented 3 years ago

Hi @NikTJ777 , thanks for opening this issue! This indeed is a weird issue you're seeing there and I see no way of k3d causing this by itself. So I guess your thoughts on this are correct, that they get garbage collected by Kubernetes. So unfortunately, we cannot really do anything there. However, you may want to consider using a registry instead, which should keep the image around forever :+1: Also, I'm curious if your last try way a success (eviction.minimum.reclaim)?