search

piraeusdatastore / piraeus-ha-controller

High Availability Controller for stateful workloads using storage provisioned by Piraeus
Apache License 2.0
15 stars 8 forks source link

Watch pods without label #12

Closed phoenix-bjoern closed 2 years ago

phoenix-bjoern commented 2 years ago

This is a proposal to increase the monitoring coverage of the HA controller, which currently only observes pods (StatefulSets) if they have been deployed with the label linstor.csi.linbit.com/on-storage-lost: remove (or any other configured label). If you use Linstor in a default StorageClass all kinds of workloads may get deployed which will most likely not have the label for the HA controller to watch their storage's health state. In the event of a node outage the HA controller will only redeploy those workloads which have set the configured label, all others will remain in an unhealthy/unknown state.

IMHO it would be more convenient if the HA controller monitors all workloads/Statefulsets with DRBD volumes attached to it without any (manually) added label. If the discovery implementation STORK uses seems to be inappropriate, an admission controller (https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/) for create and update events, which adds the label on every workload automatically, could be an alternative. OFC this automatic discovery should be configurable to give administrators the freedom to decide which approach fits their needs best. But in order to replace STORK with the HA controller (which is definitely the faster and most accurate option) an auto-discovery for relevant workloads is a "must-have" feature.

WanzenBug commented 2 years ago

In general, I agree.

It wasn't made the default until now, because we were unsure if the HA controller is in usable state. What we noticed is that in a lot of situations the HA controller loses track of the cluster state, and when it recovers it is very hesitant to actually enable the fail-over process. So while it won't suddenly start killing all your pods, it misses a lot of situations in which it actually should kill the pods, but just doesn't.

This is a bit of an architecture problem, which would need to be reworked so that the HA Controller is actually split in two, a node agent which listens for DRBD events directly and the actual HA controller which gather the cluster state via some mechanism. But that is for another issue.

I believe if you want to enable watching of all pods right now, you can do so:

--pod-label-selector=""

should make the controller watch all pods.

WanzenBug commented 2 years ago

Version 1.0.0 moved from opt-in to opt-out: Unless your pod is annotated with drbd.linbit.com/ignore-fail-over: "", it will be monitored and failed-over if necessary