search

kubernetes / kube-state-metrics

Add-on agent to generate and expose cluster-level metrics.
https://kubernetes.io/docs/concepts/cluster-administration/kube-state-metrics/
Apache License 2.0
5.21k stars 1.93k forks source link

fix: fix the issue that duplicated CR metrics are generated when CRD is deleted/applied multiple times #2257

Open CatherineF-dev opened 7 months ago

CatherineF-dev commented 7 months ago

What this PR does / why we need it: https://github.com/kubernetes/kube-state-metrics/issues/2223

# It returns same metrics data many times when CRD is deleted/applied multiple times
 curl localhost:8089/metrics
# HELP cr_creationtimestamp
# TYPE cr_creationtimestamp gauge
cr_creationtimestamp{customresource_group="example.com",customresource_kind="Bar",customresource_version="v1",name="mybar"} 1.701828773e+09
# HELP cr_resourceversion
# TYPE cr_resourceversion gauge
cr_resourceversion{customresource_group="example.com",customresource_kind="Bar",customresource_version="v1",name="mybar"} 909919
# HELP cr_creationtimestamp
# TYPE cr_creationtimestamp gauge
cr_creationtimestamp{customresource_group="example.com",customresource_kind="Bar",customresource_version="v1",name="mybar"} 1.701828773e+09
# HELP cr_resourceversion
# TYPE cr_resourceversion gauge
cr_resourceversion{customresource_group="example.com",customresource_kind="Bar",customresource_version="v1",name="mybar"} 909919

How does this change affect the cardinality of KSM: (increases, decreases or does not change cardinality)

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Mitigation for https://github.com/kubernetes/kube-state-metrics/issues/2223

k8s-ci-robot commented 7 months ago

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: CatherineF-dev Once this PR has been reviewed and has the lgtm label, please assign mrueg for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/kubernetes/kube-state-metrics/blob/main/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
dgrisonnet commented 7 months ago

I don't think this will fix the inherent issue. It seems that the mechanism detecting the deletion of CRD matched by the ksm config is not working properly. Normally when that happens, the internal store should be deleted alongside its informer because we don't want to expose metrics about this resource anymore.

dgrisonnet commented 7 months ago

To be clearer, the scenario where a specific resource appears twice in ksm code should never happen, so we should understand why it happened and fix it instead of enforcing uniqueness.

CatherineF-dev commented 6 months ago

got it, agree. Will debug again to find root cause.

CatherineF-dev commented 6 months ago

It should be related to https://github.com/kubernetes/kube-state-metrics/pull/1851 where adding PollForCacheUpdates function.

CatherineF-dev commented 6 months ago

It deletes removed CR here https://github.com/rexagod/kube-state-metrics/blob/main/internal/discovery/discovery.go#L42C24-L116, while didn't delete removed CR here https://github.com/rexagod/kube-state-metrics/blob/25a1d8da057cf761d614c59a52785335d34082d1/internal/discovery/discovery.go#L232.

How about creating a new issue and merging this PR as a mitigation? I think it needs some changes around https://github.com/kubernetes/kube-state-metrics/pull/1851.

Or we only want the fix PR instead of mitigation.

cc @dgrisonnet

rexagod commented 5 months ago

I second https://github.com/kubernetes/kube-state-metrics/pull/2257#issuecomment-1842797358, and as for https://github.com/kubernetes/kube-state-metrics/pull/2257#issuecomment-1853814510, I'd prefer the changes in this PR (instead of mitigating the issue, we want to identify the root cause and fix that since duplicacy shouldn't ideally be exhibited at all).

korjek commented 4 months ago

any updates here? It leads to losing the metrics, which is not good.

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 days ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten