carlosedp / cluster-monitoring

Cluster monitoring stack for clusters based on Prometheus Operator
MIT License
740 stars 201 forks source link

MetalLB exporter still doesn't have RBAC set up properly #98

Closed jontg closed 3 years ago

jontg commented 3 years ago

Describe the bug After deploying MetalLB using helm:

kubectl create namespace metallb-system
helm install metallb bitnami/metallb --namespace metallb-system -f metallb/metallb.values.yaml

And deploying cluster-monitor with the metallb flag enabled, I was expecting to see metrics from MetalLB flowing into prometheus. Instead, I see a number of errors from prometheus about being unable to do anything in the metallb-system namespace (based on the assumption baked into code here):

msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"metallb-system\""
msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"metallb-system\""
msg="/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"metallb-system\""

Troubleshooting

It looks like the following change to manifests/prometheus-clusterRole.yaml makes the errors go away:

+- apiGroups:
+  - ""
+  resources:
+  - pods
+  - services
+  - endpoints
+  verbs:
+  - get
+  - list
+  - watch

Action Requested

I'd like to take a crack at fixing this "the right way," but would love some help figuring out what the most modular approach should be. It looks like the clusterRole comes from a vendored dependency, so instead I propose we remove the metallb-exporter service account (which does not appear to be used), and instead bind the metallb-exporter cluster role directly to prometheus.

jontg commented 3 years ago

A proposed solution can be find in https://github.com/carlosedp/cluster-monitoring/pull/99.