search

kubernetes / kube-state-metrics

Add-on agent to generate and expose cluster-level metrics.
https://kubernetes.io/docs/concepts/cluster-administration/kube-state-metrics/
Apache License 2.0
5.36k stars 2k forks source link

kube_namespace_labels not exported after upgrading from 2.9.2 to 2.10.1 #2297

Closed jlcrow closed 7 months ago

jlcrow commented 8 months ago

I noticed the two closed issues around kube_namespace_labels, but this doesn't seem related as this was going from 2.9 to 2.10 and I haven't changed anything, just performing a helm upgrade. I had the same behavior across 8 clusters, 2 running on AKS and 6 running on GKE. The metric just disappeared, other metrics seemed fine, there were 2 other kube_namespace metrics that continued to be published, kube_namespace_labels was missing from all clusters, causing dashboards to fail.

What happened: kube_namespace_labels metric no longer present in 2.10.1

What you expected to happen: Expected the metric to continue working in minor release

How to reproduce it (as minimally and precisely as possible): I used version kube-state-metrics-5.15.3 of the prometheus-community/kube-state-metrics helm chart to upgrade kube-state-metrics using the default values, which has all metrics available. 

helm upgrade --install kube-state-metrics prometheus-community/kube-state-metrics --namespace monitoring

Anything else we need to know?:

Environment:

dashpole commented 8 months ago

/assign @rexagod @dgrisonnet /triage accepted

rexagod commented 8 months ago

I believe this is related to https://github.com/kubernetes/kube-state-metrics/pull/2190/files#diff-25a954ca6112145b3cd604fe01171ad987a40658039c5d780da990520502a72b, however, https://github.com/kubernetes/kube-state-metrics/blob/6de105ebbe0eeb5d97d9e6adf1cc83314434e8cc/internal/store/namespace_test.go#L38 is in place to catch such an issue.

@jlcrow Can you check if any other metrics besides this one are missing as well? I don't think this should be the case in v2.10.1.

jlcrow commented 8 months ago

v2.10.1 was the version I had the problem with, I'm now running v2.9.2, 2.10.1 definitely has this issue.

rexagod commented 8 months ago

ACK, are any other metrics besides this missing for you as well? Since the change affected other areas: https://github.com/kubernetes/kube-state-metrics/pull/2190/files#diff-25a954ca6112145b3cd604fe01171ad987a40658039c5d780da990520502a72b (PVs, CSRs, etc.)?

jlcrow commented 8 months ago

I wasn't able to confirm other metrics so not confident that others aren't missing, this one was just the most noticeable since we have a lot of dashboards that drive off of namespace

joeunlog commented 8 months ago

I have similar issue. There is no resource label metrics (kube_pod_labels, kube_deployment_labels, kube_statefulset_labels, etc.). And i think it's about --metric-labels-allowlist I have many cluster and set up two different version of kube-state-metrics ( 2.9.2 , 2.10.0 ). At 2.9.2 version, there is label metrics. But at 2.10.0 version, there is no label metrics. So i test to set arg --metric-labels-allowlist=nodes=[eks.amazonaws.com/nodegroup],pods=[*], and now i can get kube_node_labels and kube_pod_labels.

joeunlog commented 8 months ago

OK I found this issue https://github.com/kubernetes/kube-state-metrics/issues/1829 And check changelog https://github.com/kubernetes/kube-state-metrics/blob/main/CHANGELOG.md#v2100--2023-08-31