dweomer
commented
2 years ago 6G root disk (where your imagefs lives) is kinda small. Since we're talking libvirt here ... can you grow it? Additionally, take a look at https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ for some --kubelet-arg overrides you might consider setting:
--image-gc-high-threshold int32Default: 85 The percent of disk usage after which image garbage collection is always run. Values must be within the range [0, 100], To disable image garbage collection, set to 100. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--image-gc-low-threshold int32Default: 80 The percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. Values must be within the range [0, 100] and should not be larger than that of --image-gc-high-threshold. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--eviction-hard map[String][String]Default:
imagefs.available<15%,memory.available<100Mi,nodefs.available<10%A set of eviction thresholds (e.g. memory.available<1Gi) that if met would trigger a pod eviction. On a Linux node, the default value also includes nodefs.inodesFree<5%. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--eviction-max-pod-grace-period int32Maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met. If negative, defer to pod specified value. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--eviction-minimum-reclaim map[String][String]A set of minimum reclaims (e.g. imagefs.available=2Gi) that describes the minimum amount of resource the kubelet will reclaim when performing a pod eviction if that resource is under pressure. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--eviction-pressure-transition-period durationDefault:
5m0sDuration for which the kubelet has to wait before transitioning out of an eviction pressure condition. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--eviction-soft map[String][String]A set of eviction thresholds (e.g. memory.available<1.5Gi) that if met over a corresponding grace period would trigger a pod eviction. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
--eviction-soft-grace-period map[String][String]A set of eviction grace periods (e.g. memory.available=1m30s) that correspond to how long a soft eviction threshold must hold before triggering a pod eviction. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
vdboor
max-wittig
ClapicaVSGon
Version (k3OS / kernel)
k3os version v0.20.11-k3s1r15.4.0-84-generic #94 SMP Sun Sep 19 04:06:53 UTC 2021onx86_64Describe the bug After a recent upgrade pods could not be created. They became "Pending" due to "disk pressure" on the k3os node. However, none of the disks appeared to be full. Restarting the node temporary fixed the problem. Yet the following events are constantly posted in the kubernetes node (via
kubectl describe node):Posted every 5 minutes in
/var/log/k3s-service.log:so some kind of garbage collection tries to run, but there is nothing to collect despite what the message claims.
Additional context
Using
crictl imagesdoes not show unused images,crictl rmi --prunedoesn't release any images at all.The only odd observation I have is that
crictl imagesshows various containers with a<none>tag. This happens when theimage:tag@sha265:..format is used. This happens for example withk8s.gcr.io/ingress-nginx/controller:v1.0.1@sha256:26bbd57f32bac3b30f90373005ef669aae324a4de4c19588a13ddba399c6664eOutput from
df -h | sort:The K3OS machine runs inside libvirt.