Pod readiness metrics, part 2

erhudy commented 7 months ago

What would you like to be added:

I would like kube-state-metrics to be able to report on pod readiness gates in order to have proper alerting set up for readiness gates which are not proceeding.

Why is this needed:

This is to be able to monitor and alert on stuck readiness gates from within the viewpoint of Kubernetes.

This issue was originally raised in https://github.com/kubernetes/kube-state-metrics/issues/1981 with a proposed solution (combining it into the overall pod ready status), but was closed because that solution was deemed not satisfactory. I propose an alternative solution below.

Describe the solution you'd like

I propose the creation of two new metrics, tentatively named kube_pod_readiness_gate_status_total and kube_pod_readiness_gate_status_ready, to track the number of total PRGs and the ready number thereof.

The disposition of these metrics can be determined fairly simply by inspecting the pod spec and status (examples taken below from a real pod):

spec:
  readinessGates:
  - conditionType: target-health.elbv2.k8s.aws/k8s-kongk8s-kongk8su-928018d45d
  - conditionType: target-health.elbv2.k8s.aws/k8s-kongk8s-kongk8su-01c4e87d17
  - conditionType: target-health.elbv2.k8s.aws/k8s-kongk8s-kongk8su-02d5a84fe7
  - conditionType: target-health.elbv2.k8s.aws/k8s-kongk8s-kongk8su-15c742bed9

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: null
    message: 'Health checks failed with these codes: [401]'
    reason: Target.ResponseCodeMismatch
    status: "True"
    type: target-health.elbv2.k8s.aws/k8s-kongk8s-kongk8su-15c742bed9
  - lastProbeTime: null
    lastTransitionTime: null
    message: 'Health checks failed with these codes: [401]'
    reason: Target.ResponseCodeMismatch
    status: "True"
    type: target-health.elbv2.k8s.aws/k8s-kongk8s-kongk8su-928018d45d
  - lastProbeTime: null
    lastTransitionTime: null
    message: Target registration is in progress
    reason: Elb.RegistrationInProgress
    status: "True"
    type: target-health.elbv2.k8s.aws/k8s-kongk8s-kongk8su-01c4e87d17
  - lastProbeTime: null
    lastTransitionTime: null
    message: Target registration is in progress
    reason: Elb.RegistrationInProgress
    status: "True"
    type: target-health.elbv2.k8s.aws/k8s-kongk8s-kongk8su-02d5a84fe7

For a given pod, the total metric would be determined by the number of entries in the spec.readinessGates array, and the number of ready PRGs would be determined by following each entry from the array to the matching conditions of that type in status.conditions and seeing which ones are reporting status: "True".

Additional context

I am happy to do the work to implement this, I just want a thumbs up on the proposed methodology before I do anything.

dgrisonnet commented 7 months ago

/assign /triage accepted

dgrisonnet commented 7 months ago

These metrics are going to be quite expensive.

Is your goal to monitor the elb readiness via the pods? Because for pod readiness, you should already be able to alert on kube_pod_status_ready.

erhudy commented 7 months ago

Pod readiness doesn't necessarily capture what I'm looking for here. It's possible for Kubernetes to consider a pod ready, but have pod readiness gates failing; e.g. a misconfiguration on an associated ingress could prevent the AWS LB controller from provisioning an ELB, which would be reflected on the pod readiness gate, but the pod itself would still show up as N/N ready because it's fine from the kubelet's perspective.

kubernetes / kube-state-metrics

Pod readiness metrics, part 2 #2250