kubernetes-sigs / metrics-server

Scalable and efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines.
https://kubernetes.io/docs/tasks/debug-application-cluster/resource-metrics-pipeline/
Apache License 2.0
5.72k stars 1.86k forks source link

default components.yaml for 1.24+ k8s clusters lacks ServiceAccount token/secret creation #1451

Closed drook closed 2 weeks ago

drook commented 5 months ago

What happened: After the installation with the default components.yaml scenario the metrics-server is unable to scrape the kubelet metrics.

What you expected to happen: After the installation with the default components.yaml scenario the metrics-server is able to scrape the kubelet metrics.

Anything else we need to know?: Since 1.24+ version the token/secret for the ServiceAccount is no longer automatically created, thus metrics-server is getting loads of the errors when trying to scrape the kubelet API on port 10250:

I0325 10:23:07.712822       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
E0325 10:23:15.451208       1 scraper.go:140] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="k8s-ctl3.foo.bar"
E0325 10:23:15.457767       1 scraper.go:140] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="k8s-wrk1.foo.bar"
E0325 10:23:15.470400       1 scraper.go:140] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="k8s-wrk3.foo.bar"
E0325 10:23:15.480009       1 scraper.go:140] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="k8s-wrk4.foo.bar"
E0325 10:23:15.498523       1 scraper.go:140] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="k8s-wrk2.foo.bar"
E0325 10:23:15.504590       1 scraper.go:140] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="k8s-ctl1.foo.bar"
E0325 10:23:15.504832       1 scraper.go:140] "Failed to scrape node" err="request failed, status: \"403 Forbidden\"" node="k8s-ctl2.foo.bar"

Environment:

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"", GitTreeState:"", BuildDate:"2023-02-10T12:43:36Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"freebsd/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6", GitCommit:"ff2c119726cc1f8926fb0585c74b25921e866a28", GitTreeState:"clean", BuildDate:"2023-01-18T19:15:26Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

bundled one for 0.7.0 (also tried downgrades til 0.6.1, before realizing this is not the real issue.)

(don't think the rest is useful).

P.S. Shame, but I'm still unable to figure out how to get past 403 error even with the secret/token created manually.

/kind bug

k8s-ci-robot commented 5 months ago

This issue is currently awaiting triage.

If metrics-server contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
LKD-PIX commented 3 months ago

i am having the exact same issue

LKD-PIX commented 3 months ago

Were you able to solve it?

logicalhan commented 2 weeks ago

/close 1.24 is no longer a supported kubernetes version.

k8s-ci-robot commented 2 weeks ago

@logicalhan: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/metrics-server/issues/1451#issuecomment-2305227732): >/close >1.24 is no longer a supported kubernetes version. Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.