And deploying cluster-monitor with the metallb flag enabled, I was expecting to see metrics from MetalLB flowing into prometheus. Instead, I see a number of errors from prometheus about being unable to do anything in the metallb-system namespace (based on the assumption baked into code here):
msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"metallb-system\""
msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"metallb-system\""
msg="/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"metallb-system\""
Troubleshooting
It looks like the following change to manifests/prometheus-clusterRole.yaml makes the errors go away:
I'd like to take a crack at fixing this "the right way," but would love some help figuring out what the most modular approach should be. It looks like the clusterRole comes from a vendored dependency, so instead I propose we remove the metallb-exporter service account (which does not appear to be used), and instead bind the metallb-exporter cluster role directly to prometheus.
Describe the bug After deploying MetalLB using helm:
And deploying cluster-monitor with the metallb flag enabled, I was expecting to see metrics from MetalLB flowing into prometheus. Instead, I see a number of errors from prometheus about being unable to do anything in the
metallb-system
namespace (based on the assumption baked into code here):Troubleshooting
It looks like the following change to
manifests/prometheus-clusterRole.yaml
makes the errors go away:Action Requested
I'd like to take a crack at fixing this "the right way," but would love some help figuring out what the most modular approach should be. It looks like the clusterRole comes from a vendored dependency, so instead I propose we remove the
metallb-exporter
service account (which does not appear to be used), and instead bind themetallb-exporter
cluster role directly to prometheus.