search

geerlingguy / turing-pi-cluster

DEPRECATED - Turing Pi cluster configuration for Raspberry Pi Compute Modules
https://www.youtube.com/watch?v=kgVz4-SEhbE
MIT License
355 stars 47 forks source link

K3s spitting out lots of logs from metadatainformer/informer #6

Closed geerlingguy closed 4 years ago

geerlingguy commented 4 years ago

K3s is spitting out a log message like the following, five times per second. The master node is stuck at a load of 1.5-2.0:

May 23 20:29:53 turing-master k3s[334]: E0523 20:29:53.450194 334 reflector.go:153] k8s.io/client-go/metadata/metadatainformer/informer.go:89: Failed to list *v1.PartialObjectMetadata: the server could not find the requested resource

See related issue: https://github.com/rancher/k3s/issues/1259

geerlingguy commented 4 years ago

Looks like this is the root cause; a Kubernetes bug: https://github.com/kubernetes/kubernetes/issues/79610

geerlingguy commented 4 years ago

The fix?

cd k3s-ansible-master
ansible-playbook reset.yml -i inventory/hosts.ini
ansible all -i inventory/hosts.ini -m reboot -b
ansible-playbook reset.yml -i inventory/hosts.ini
ansible-playbook site.yml -i inventory/hosts.ini

Basically, delete the cluster and build it fresh. Ugh, that's not a fun way to live!