Closed SpComb closed 6 years ago
This seems to be covered by https://github.com/kubernetes/kubernetes/issues/31345, with a new RetryWatcher
in https://github.com/kubernetes/kubernetes/pull/50102... that seems to retry the Watch
for any and all errors, but it does that by tracking the lastModifiedVersion
, so that it doesn't need to retry the Get
...
Retrying the Get
for arbitrary errors doesn't seem like a good idea, because then it will get stuck on persistent errors like missing ClusterRole rules...
Thinking about it more, it would make sense for the top-level Kube.AcquireLock()
to retry the locking regardless of any kube errors... that implementation would be related to #1 as a high-level functionality, so with --schedule-window=2h
it should retry for up to two hours.
If a kube master host upgrades and reboots, it will interrupt any simultaneously scheduled worker hosts upgrades, because the kube API watch for the upgrade lock will fail:
The host-upgrades pod will be restarted, but it will miss that scheduled upgrade. With simultaneously scheduled daily upgrades on both master and worker nodes, the worker nodes may thus only install their upgrades the following day, depending on the randomized ordering of whether the master node(s) happen to acquire their locks first.
The kube locking should probably retry the watch - for a HA setup this should immediately succeed, but for a single-master setup, the retry needs to be patient enough to allow the master node to complete its reboot.