search

kubernetes / kube-state-metrics

Add-on agent to generate and expose cluster-level metrics.
https://kubernetes.io/docs/concepts/cluster-administration/kube-state-metrics/
Apache License 2.0
5.34k stars 1.99k forks source link

Incompatible changes about custom resource metrics #2013

Open Garrybest opened 1 year ago

Garrybest commented 1 year ago

I have introduced a feature in #1644, the intent about this interface is to allow users build their own repos and make KSM as a dependency library in their go.mod. So all the custom metrics could be customized by implement these predefined interfaces and pass the factories to run KSM.

After that PR, #1710 is contributed for collecting custom metrics by merely providing a config without building their own repo. It's a great step. I believe users could leverge KSM by form of dependency library or direct config.

Now, according to https://github.com/kubernetes/kube-state-metrics/pull/1928#discussion_r1115433085, this PR has already removed this parameter factories ...customresource.RegistryFactory. So we could never leverage KSM as a dependency library. I believe some users have already done this but now it's incompatible, e.g., https://github.com/kubernetes/kube-state-metrics/pull/1644#issuecomment-990892061. So I suggest we move back to adding this parameter in server.go again.

CatherineF-dev commented 1 year ago

Could you provide a simple example which makes KSM as a library? And it couldn't work with latest KSM version.

Trying to reproduce this issue.

Garrybest commented 1 year ago

Hi @CatherineF-dev. I introduce #1644 and want to extend KSM by using it as a dependency library. So we could build our own repo like this. Here is my case, this code belongs to a private library.

package main

import (
    "fmt"
    "os"

    "github.com/prometheus/common/version"
    apiserver "k8s.io/apiserver/pkg/server"
    "k8s.io/klog/v2"
    "k8s.io/kube-state-metrics/v2/pkg/app"
    "k8s.io/kube-state-metrics/v2/pkg/customresource"
    "k8s.io/kube-state-metrics/v2/pkg/options"

    "git.woa.com/c.oa.com/karmada-state-metrics/pkg/karmada"
)

var (
    // This is a user-defined custom resources factory which could be passed to KSM
    // and be initialized by KSM custom resource framework.
    factories = []customresource.RegistryFactory{
        new(karmada.ClusterFactory),
        new(karmada.ResourceBindingFactory),
    }
)

func main() {
    opts := options.NewOptions()
    opts.AddFlags()

    if err := opts.Parse(); err != nil {
        klog.Fatalf("Parsing flag definitions error: %v", err)
    }

    if opts.Version {
        fmt.Printf("%s\n", version.Print("kube-state-metrics"))
        os.Exit(0)
    }

    if opts.Help {
        opts.Usage()
        os.Exit(0)
    }

    ctx :=apiserver.SetupSignalContext()
    // We pass these factories to KSM and compile our own factory with KSM internal metrics.
    if err := app.RunKubeStateMetrics(ctx, opts, factories...); err != nil {
        klog.Fatalf("Failed to run kube-state-metrics: %v", err)
    }
}

We build our own structs by implementing customresource.RegistryFactory and pass them to app.RunKubeStateMetrics, so KSM could initialize metrics as well as our custom metrics. Now according to https://github.com/kubernetes/kube-state-metrics/pull/1928#discussion_r1115433085, parameters of app.RunKubeStateMetrics have been changed so the code above cannot be compiled.

CatherineF-dev commented 1 year ago

QQ: got it. Why do you want to use KSM as library instead of deploying the KSM agent?

Is it because you only want customresource feature and want to reduce memory usage

Garrybest commented 1 year ago

QQ: got it. Why do you want to use KSM as library instead of deploying the KSM agent?

Is it because you only want customresource feature and want to reduce memory usage

Do you mean https://github.com/kubernetes/kube-state-metrics/blob/main/docs/customresourcestate-metrics.md? Well, I didn't use this method to expose my metrics because these metrics have a complex logic to calculate. So custom resource state metrics cannot meet my demand.

The initial goal of #1644 is to extend KSM as a library for some complex metrics exposing. It is similar with framework of kube-scheduler scheduler-plugins as it does allow users to build some high-level extension.

So inspired by scheduler-plugins, we could add the parameters back and let users introduce their out-of-tree RegistryFactory.

mrueg commented 1 year ago

In my personal opinion kube-state-metrics is not meant to be used as a library and rather a standalone deployment. I also don't know if the team has the desire and capacity to support this use case. Have you looked at https://github.com/kubernetes-sigs/custom-metrics-apiserver? This might be a viable option for you.

dgrisonnet commented 1 year ago

/assign /triage accepted

k8s-triage-robot commented 6 months ago

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

rexagod commented 2 months ago

/remove-lifecycle rotten /triage accepted /assign

AFAICT KSM never explicitly supported stable library APIs, but nonetheless, they may still be in use by the community. I believe this needs some more thought.

@Garrybest Would you be available for bringing this up in next SIG call?