search

kontena / pharos-cluster

Pharos - The Kubernetes Distribution
https://k8spharos.dev/
Apache License 2.0
311 stars 43 forks source link

Kube-dns service is broken after upgrade to 1.3.0-rc.4 #577

Closed jakolehm closed 6 years ago

jakolehm commented 6 years ago

Basically it cannot find matching pods -> endpoints are empty.

Saw this in controller-manager logs:

W0829 13:41:11.390017       1 endpoints_controller.go:380] Dropping service "kube-system/kube-dns" out of the queue: Endpoints "kube-dns" is invalid: subsets[0].addresses[0].nodeName: Forbidden: Cannot change NodeName for 172.17.0.2 to jari-pharos-worker-1
E0829 13:41:11.390035       1 endpoints_controller.go:382] Endpoints "kube-dns" is invalid: subsets[0].addresses[0].nodeName: Forbidden: Cannot change NodeName for 172.17.0.2 to jari-pharos-worker-1

I managed to fix it by editing service and saving it (kubectl edit svc ...).

jakolehm commented 6 years ago

It seems that this does not happen every time (tested upgrade with another cluster).

jakolehm commented 6 years ago

DNS is still broken even if service endpoints are there.

jakolehm commented 6 years ago

Upgrade seems to work ok with cri-o. Maybe this is docker related issue?

jakolehm commented 6 years ago

Root cause seems to be missing cni config for kubelet (on upgrade).