search

kubernetes / kube-state-metrics

Add-on agent to generate and expose cluster-level metrics.
https://kubernetes.io/docs/concepts/cluster-administration/kube-state-metrics/
Apache License 2.0
5.44k stars 2.02k forks source link

When using limited privilege environment (roles) empty metrics endpoint #2400

Open SennaSemakula opened 5 months ago

SennaSemakula commented 5 months ago

What happened: I ran kube-state-metrics using Roles and RoleBindings following: https://github.com/kubernetes/kube-state-metrics?tab=readme-ov-file#limited-privileges-environment but I did not pass in --namespaces as it should default to all namespaces including the current one. When I navigate to localhost:8080/metrics the endpoint is blank.

Screenshot 2024-05-24 at 15 48 02

What you expected to happen: When using limited privileges (roles and not clusterroles) it should have metrics on the current namespace. Note this works if you do pass in --namespaces but it should work regardless as the default value monitors all namespaces based on: https://github.com/kubernetes/kube-state-metrics/blob/main/pkg/options/options.go#L146

How to reproduce it (as minimally and precisely as possible):

  1. Follow steps but make sure you do not specify --namespaces as it should default to all namespaces by default
  2. Navigate to localhost:8080/metrics and you should see no metrics

Anything else we need to know?: I'm working on deploying kube-state-metrics in a tenanted environment that is using: https://github.com/kubernetes-sigs/hierarchical-namespaces.

Environment:

thunderbird86 commented 5 months ago

Have the same issue, but I didn't setup any limited privileges environment, it works fine for some time and now it has the same empty responce

Serializator commented 4 months ago

The problem originates from the use of v1.NamespaceAll, where KSM assumes it can list and watch resources at the cluster scope. This is a wrong assumption in a limited privilege environment.

W0623 13:07:43.643600  104637 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:kube-state-metrics" cannot list resource "pods" in API group "" at the cluster scope

https://github.com/kubernetes/kube-state-metrics/blob/main/internal/store/builder.go#L514-L524

The thing is though, without the permission to list namespaces, KSM can't determine what namespaces we have access to either. The minimal requirement in a limited privilege environment for something like this to work is a cluster role (binding) to allow the listing of namespaces across the cluster.

dgrisonnet commented 3 months ago

/assign @mrueg /triage accepted