search

kubernetes / kube-state-metrics

Add-on agent to generate and expose cluster-level metrics.
https://kubernetes.io/docs/concepts/cluster-administration/kube-state-metrics/
Apache License 2.0
5.36k stars 2k forks source link

Kube Node Status NotReady detection #2345

Open rgarcia89 opened 6 months ago

rgarcia89 commented 6 months ago

What would you like to be added: Currently the node state metrics is lagging a possibility to detect if a node has become notReady for any specific reasons. I would therefore like to request creation of a metric like node ready seconds for example or last status chang in order to be able to detect such situations.

rgarcia89 commented 6 months ago

@dgrisonnet @logicalhan

ricardoapl commented 4 months ago

I would be happy to submit a patch to support your use case.

However, I noticed we already have kube_node_status_condition with labels for condition and status. Doesn't that already solve your problem?

Perhaps querying for something like kube_node_status_condition{condition="Ready", status!="true"}

https://github.com/kubernetes/kube-state-metrics/blob/main/docs/metrics/cluster/node-metrics.md

rgarcia89 commented 4 months ago

@ricardoapl this only provides the status for the specific point in time when you scraped it. However, what if you scrape every 30 seconds, and within that interval, the node becomes NotReady for 10 seconds? You would miss that status change.

From the idea it could be comparable with the kube_pod_container_status_last_terminated_reason metric.

dgrisonnet commented 4 months ago

I don't think we can get that information from the NodeStatus today: https://github.com/kubernetes/api/blob/v0.30.1/core/v1/types.go#L5871-L5936

Also if you miss the status it most likely means that it auto resolved in less than 30 seconds, so I am not sure how useful would be the information.

rgarcia89 commented 4 months ago

@dgrisonnet I faced an issue with some nodes that switched to NotReady state which caused issues for some pods that I cannot recall anymore. Unfortunately the status change of the metric was not recorded by any metric. Due to that I have created an alert on log entries which is making us aware nowadays.

I had a conversation with one of the maintainers during KubeCon Paris which was also of the opinion that this metric is missing. I cannot recall his name unfortunately.

However if the API does not provide any way to obtain this data things will become complicated indeed.

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

dashpole commented 3 weeks ago

/assign @CatherineF-dev /triage accepted