k0sproject / k0s

k0s - The Zero Friction Kubernetes
https://docs.k0sproject.io
Other
3.74k stars 365 forks source link

When leader lease is lost applier manager is not restarted #5122

Open emosbaugh opened 3 weeks ago

emosbaugh commented 3 weeks ago

Before creating an issue, make sure you've checked the following:

Platform

No response

Version

v1.29.9+k0s

Sysinfo

`k0s sysinfo`
➡️ Please replace this text with the output of `k0s sysinfo`. ⬅️

What happened?

When a third controller is added the leader lease is somehow lost and when it is re-acquired the applier-manager is not restarted resulting in updates to manifests or stacks not being applied.

Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: I1015 16:55:27.386215    3978 leaderelection.go:285] failed to renew lease kube-node-lease/k0s-endpoint-reconciler: timed out waiting for the condition
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: I1015 16:55:27.386295    3978 leaderelection.go:285] failed to renew lease kube-node-lease/k0s-ctrl-node-e3a0d-00: timed out waiting for the condition
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Lost leader lease" component=controllerlease
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: I1015 16:55:27.391034    3978 leaderelection.go:250] attempting to acquire leader lease kube-node-lease/k0s-ctrl-node-e3a0d-00...
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Lost leader lease" component=poolleaderelector
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: I1015 16:55:27.391062    3978 leaderelection.go:250] attempting to acquire leader lease kube-node-lease/k0s-endpoint-reconciler...
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="lost leader lease" component=poolleaderelector
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=error msg="lost leader lease, this should not really happen!?!?!?" component=controllerlease

...

Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="manifest watcher done" component=applier-manager

...

Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Acquired leader lease" component=poolleaderelector
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="acquired leader lease" component=poolleaderelector
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Acquired leader lease" component=controllerlease
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="acquired leader lease" component=controllerlease
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Acquired leader lease" component=extensions_controller
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Acquired leader lease" component=extensions_controller

...

Steps to reproduce

1. 2. 3.

Expected behavior

Changes to manifests dir will continue to be applied to the cluster

Actual behavior

Changes are no longer reflected in the cluster.

Screenshots and logs

k0scontroller-logs.txt k0scontroller-logs.txt k0scontroller-logs.txt

Additional context

No response

emosbaugh commented 3 weeks ago

Fixed by https://github.com/k0sproject/k0s/pull/5062 once merged