search

grafana / mimir

Grafana Mimir provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.
https://grafana.com/oss/mimir/
GNU Affero General Public License v3.0
3.98k stars 503 forks source link

Mimir grafana/Kube-state-metrics #6368

Open rodrigoccaldas opened 10 months ago

rodrigoccaldas commented 10 months ago

Discussed in https://github.com/grafana/mimir/discussions/6367

Originally posted by **rodrigoccaldas23** October 13, 2023 Hello, We are facing some issues when trying to have our Kubernetes dashboards with metrics on grafana mimir. We already have KSM helmrelease (chart v5.7.0) installed on the cluster with the following values: **KSM values** ``` namespaceOverride: "" rbac: create: true releaseLabel: true prometheus: monitor: enabled: true interval: "" sampleLimit: 0 targetLimit: 0 labelLimit: 0 labelNameLengthLimit: 0 labelValueLengthLimit: 0 scrapeTimeout: "" proxyUrl: "" honorLabels: true metricRelabelings: [] relabelings: [] selfMonitor: enabled: false ``` **Grafana mimir datasource:** ``` datasources: datasources.yaml: apiVersion: 1 datasources: - name: Mimir type: prometheus url: http://mimir-nginx.mimir.svc.cluster.local/prometheus isDefault: true jsonData: httpHeaderName1: 'X-Scope-OrgID' secureJsonData: httpHeaderValue1: 'metamonitoring' ``` **Mimir metamonitoring values:** ``` metaMonitoring: dashboards: enabled: true annotations: k8s-sidecar-target-directory: "/tmp/dashboards/mimir_dashboard" grafana_folder: "mimir" labels: grafana_dashboard: "1" serviceMonitor: enabled: true grafanaAgent: enabled: true imageRepo: configReloader: repo: quay.io image: prometheus-operator/prometheus-config-reloader tag: v0.47.0 grafanaAgent: repo: docker.io image: grafana/agent tag: v0.29.0 installOperator: true metrics: scrapeK8s: enabled: true kubeStateMetrics: namespace: kube-system labelSelectors: app.kubernetes.io/name: kube-state-metrics ``` image Mimir chart version ``` chart: mimir-distributed version: "5.1.0" ``` Also, we are able to validate metrics when port-forwarding KSM pod: image Any ideia why we are getting no data on k8s grafana dashboards? Is the grafana agent installed with metamonitoring enough or do we need to install another one in the same cluster?
luong-komorebi commented 5 months ago

I am also seeing this issues. Some metrics such as kube_deployment_status_condition does not appear in mimir albeing being properly exposed in kube-state-metrics

luong-komorebi commented 5 months ago

@rodrigoccaldas23 I figured out the reason. This chart is using grafana operator and created grafana agent accordingly. Grafana agent by default will not allow all metrics from kube-state-metrics. You have to turn off the default allow list so that all your metrics are consumed, or you have to explicitly define them.

Document https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/manage-configuration/#kube-state-metrics

The file where grafana agent specs are created https://github.com/grafana/mimir/blob/main/operations/helm/charts/mimir-distributed/templates/metamonitoring/grafana-agent.yaml and here ideally it would have to allow us to pass it some extra configurations for the allowlist. However, that right now is not the case so you would have to open a pull request for that or rely on other methods of installing grafana agent (not using the default one by this chart) and then turn the allowlist off or explictly define