search

grafana / kubernetes-app

A set of dashboards and panels for kubernetes.
https://grafana.com/plugins/grafana-kubernetes-app
Apache License 2.0
401 stars 149 forks source link

No Data Points Collected After Successful Deployment? #74

Open zimmertr opened 4 years ago

zimmertr commented 4 years ago

Hello, I have recently installed, enabled, and configured this app. As you can see here, the Cluster Info is properly populated with information scraped from my cluster. Additionally, the generated data source, as shown here, indicates that the configuration is working properly.

However, as seen here, no Data Points are being collected by Grafana for any other automatically generated Dashboards.

I have manually modified my prometheus.yml file for the Prometheus server running on my Cluster to contain the kubernetes-kubelet Job as well as deployed the required Kube Exporter and Kube State Metrics deployment manifests, which are happily running on my cluster right now.

Is there something else I need to do? The cluster is configured using the SSL Certificates found within my ~/.kube/config file.

zimmertr commented 4 years ago

EDIT: It appears that the user associated with the certificates in my ~/.kube/config file doesn't have the required permissions to list resources in the cluster as indicated by the following example log from the kube-state-metrics pod.

k8s.io/kube-state-metrics/collectors/resourcequota.go:67: Failed to list *v1.ResourceQuota: resourcequotas is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "resourcequotas" in API group "" at the cluster scope

How should I properly create a user to remediate this?

cbluth commented 4 years ago

@zimmertr use this guide: https://docs.bitnami.com/kubernetes/how-to/configure-rbac-in-your-kubernetes-cluster/

generate a new certificate with the proper access, and itll work fine.

zimmertr commented 4 years ago

Hi @cbluth, I implemented the steps in that Bitnami guide as an Ansible playbook found here.

Here is the ClusterRole that the user uses and here is how it is bound to the user.

However, despite this, Grafana throws this error when I configure it with the generated certificates.

Furthermore, if I attempt to run any command with the user context, it says I'm unauthorized.

$> kubectl --context=grafana-prometheus-scraper get pods -n grafana
error: You must be logged in to the server (Unauthorized)

Any ideas?