search

senthilrch / kube-fledged

A kubernetes operator for creating and managing a cache of container images directly on the cluster worker nodes, so application pods start almost instantly
Apache License 2.0
1.26k stars 119 forks source link

Bug: Unable to delete image on node with containerd runtime #89

Closed senthilrch closed 3 years ago

senthilrch commented 3 years ago

Pod event:-

Events:
  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Normal   Scheduled    108s                default-scheduler  Successfully assigned kube-fledged/imagecache1-26ff5-m85lb to node-10-120-127-21
  Warning  FailedMount  45s (x8 over 108s)  kubelet            MountVolume.SetUp failed for volume "runtime-sock" : hostPath type check failed: /run/containerd/containerd.sock is not a file

Controller log:-

I0730 05:37:37.866026 1 image_manager.go:415] Job imagecache1-7ksk8 created (delete:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-25, runtime: containerd://1.4.4)
I0730 05:37:37.874927 1 image_manager.go:415] Job imagecache1-hxjnv created (delete:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-26, runtime: containerd://1.4.4)
I0730 05:37:37.887076 1 image_manager.go:415] Job imagecache1-pkq69 created (delete:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-26, runtime: containerd://1.4.4)
I0730 05:37:37.895511 1 image_manager.go:415] Job imagecache1-69qxk created (delete:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-27, runtime: containerd://1.4.4)
I0730 05:37:37.899097 1 image_manager.go:415] Job imagecache1-sctsk created (delete:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-27, runtime: containerd://1.4.4)
I0730 05:37:38.030825 1 image_manager.go:415] Job imagecache1-ss78m created (delete:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-20, runtime: containerd://1.4.4)
I0730 05:37:38.231420 1 image_manager.go:415] Job imagecache1-k7w8p created (delete:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-20, runtime: containerd://1.4.4)
I0730 05:37:38.431199 1 image_manager.go:415] Job imagecache1-6q8j8 created (delete:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-21, runtime: containerd://1.4.4)
I0730 05:37:38.630803 1 image_manager.go:415] Job imagecache1-26ff5 created (delete:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-21, runtime: containerd://1.4.4)
I0730 05:37:38.830593 1 image_manager.go:415] Job imagecache1-q82c5 created (delete:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-22, runtime: containerd://1.4.4)
I0730 05:37:39.030576 1 image_manager.go:415] Job imagecache1-6hvtv created (delete:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-22, runtime: containerd://1.4.4)
I0730 05:37:39.230910 1 image_manager.go:415] Job imagecache1-gpbfx created (delete:- quay.io/bitnami/redis:6.2.5 --> node-10-120-127-27, runtime: containerd://1.4.4)
I0730 05:37:39.431366 1 image_manager.go:415] Job imagecache1-4bnpx created (delete:- quay.io/bitnami/mariadb:10.5.11 --> node-10-120-127-27, runtime: containerd://1.4.4)
I0730 05:42:39.432314 1 image_manager.go:221] Job imagecache1-pkq69 expired (delete: quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-26)
I0730 05:42:39.432385 1 image_manager.go:221] Job imagecache1-69qxk expired (delete: quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-27)
I0730 05:42:39.432400 1 image_manager.go:221] Job imagecache1-sctsk expired (delete: quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-27)
I0730 05:42:39.432410 1 image_manager.go:221] Job imagecache1-k7w8p expired (delete: quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-20)
I0730 05:42:39.432427 1 image_manager.go:221] Job imagecache1-lfrtf expired (delete: quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-24)
I0730 05:42:39.432436 1 image_manager.go:221] Job imagecache1-7ksk8 expired (delete: quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-25)
I0730 05:42:39.432445 1 image_manager.go:221] Job imagecache1-ss78m expired (delete: quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-20)
I0730 05:42:39.432454 1 image_manager.go:221] Job imagecache1-26ff5 expired (delete: quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-21)
I0730 05:42:39.432463 1 image_manager.go:221] Job imagecache1-4d4mr expired (delete: quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-24)
I0730 05:42:39.432475 1 image_manager.go:221] Job imagecache1-k5vkd expired (delete: quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-23)
I0730 05:42:39.432483 1 image_manager.go:221] Job imagecache1-q9r4l expired (delete: quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-25)
I0730 05:42:39.432492 1 image_manager.go:221] Job imagecache1-6hvtv expired (delete: quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-22)
I0730 05:42:39.432500 1 image_manager.go:221] Job imagecache1-4bnpx expired (delete: quay.io/bitnami/mariadb:10.5.11 --> node-10-120-127-27)
I0730 05:42:39.432515 1 image_manager.go:221] Job imagecache1-tgnnf expired (delete: quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-23)
I0730 05:42:39.432524 1 image_manager.go:221] Job imagecache1-6q8j8 expired (delete: quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-21)
I0730 05:42:39.432533 1 image_manager.go:221] Job imagecache1-q82c5 expired (delete: quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-22)
I0730 05:42:39.432541 1 image_manager.go:221] Job imagecache1-gpbfx expired (delete: quay.io/bitnami/redis:6.2.5 --> node-10-120-127-27)
I0730 05:42:39.432553 1 image_manager.go:221] Job imagecache1-hxjnv expired (delete: quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-26)
I0730 05:42:41.045426 1 controller.go:430] Starting to sync image cache imagecache1(statusupdate)
I0730 05:42:41.096181 1 controller.go:664] Annotation kubefledged.io/purge-imagecache removed from imagecache(imagecache1)
I0730 05:42:41.096224 1 controller.go:633] Completed sync actions for image cache imagecache1(statusupdate)
I0730 05:42:41.096311 1 event.go:282] Event(v1.ObjectReference{Kind:"ImageCache", Namespace:"kube-fledged", Name:"imagecache1", UID:"c4b5579d-f075-40fe-aa99-4ebe73fefa63", APIVersion:"kubefledged.io/v1alpha2", ResourceVersion:"63611135", FieldPath:""}): type: 'Warning' reason: 'ImageCachePurge' Image deletion failed for some images. Please see "failures" section