Operator: Monitor pods instead of services

dasMulli commented 3 years ago

Writing down some thoughts after an analysis for our use cases. At the moment I have no immediate need for these changes but want to provide input. (If this changes, I might be able to help with an implementation)

Currently the K8s queries services by label and configures the health check ui with its ClusterIP or LoadBalancer IP as endpoint.

This has a few disadvantages:

The health check hits a random pod behind this service. This can lead to flaky health check reports when you have deployments / stateful sets with >1 replica count.
This may not work as expected with headless services for services that don't load balance but who are only used to discover the endpoint objects pointing to all stateful set instance pods. (e.g. partitioning or replication algorithm where clients discover all cluster members by querying the endpoints.)

It may be more practical to discover pods instead of services.. And maybe do this by annotations instead of labels (this is what other operators like cert-manager, elastic filebeat or ingress controllers do)

eiximenis commented 3 years ago

Hi @dasMulli Work for inspecting pods instead of services is in progress and (hopefully) will be finished soon :D

@unaizorrilla Could you assign this issue to me? Thx!

mhbuck commented 1 year ago

Hi @eiximenis I realise this is a year later - Did the pod work progress or is there a PR/branch that was looking into it that I could look into?

Xabaril / AspNetCore.Diagnostics.HealthChecks

Operator: Monitor pods instead of services #759