search

vigevenoj / k8sharkbait

Spins up a kubernetes cluster with applications on Linodes from scratch
3 stars 1 forks source link

Adjust Kubelet garbage collection and logging retention #4

Open vigevenoj opened 6 years ago

vigevenoj commented 6 years ago

Currently if a node experiences disk pressure, Kubelet GC removes infrequently-used images. Our usual case is either kube-proxy or owntracks2db. kube-proxy can be downloaded from public repositories, but owntracks2db is side-loaded.

Fixing #3 will make this less painful to recover from, but docker logs should be truncated periodically and logging should be adjusted to be less noisy. traefik is set to debug and could be changed to a less-noisy logging level as a quick win.

vigevenoj commented 6 years ago

An overview of Kubelet GC is available in https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/

vigevenoj commented 6 years ago

Temporary workaround, look at biggest log and truncate them

find /var/lib/docker/containers -name "*-json.log" -exec ls -lh {} \; | sort -h -k5

vigevenoj commented 6 years ago

We should configure the docker logging to rotate. 10 100m files should be enough to troubleshoot.