grafana / mimir

Grafana Mimir provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.
https://grafana.com/oss/mimir/
GNU Affero General Public License v3.0
4.13k stars 529 forks source link

mimir-distributed has non-working Grafana dashboards #9848

Open ripa1993 opened 1 day ago

ripa1993 commented 1 day ago

Describe the bug

Grafana dashboards delivered with the Helm Chart use metrics that reference wrong pod names. In the built mixins, the metrics assume pod names starting with mimir-[component] but in mimir-distributed chart the components are named mimir-distributed-[component]

To Reproduce

Steps to reproduce the behavior:

  1. Deploy mimir-distributed with Grafana dashboards enabled
  2. Observe Dashboards with "no data" in panels

Expected behavior

Dashboards panel should be able to fetch the correct data.

Environment

Additional Context

narqo commented 6 hours ago

In the built mixins, the metrics assume pod names starting with mimir-[component] but in mimir-distributed chart the components are named mimir-distributed-[component]

I think, with Helm chart this depends on the release name, user passes to the helm install command. That is, following Mimir Helm chart's getting started guide (note the "mimir" release name in the command above):

% helm install mimir grafana/mimir-distributed --namespace test-monitoring

This installs k8s resources with the mimir- prefix, i.e.

% kubectl -n test-monitoring get po
NAME                                        READY   STATUS      RESTARTS   AGE
mimir-compactor-0                           1/1     Running     0          8m27s
mimir-store-gateway-zone-c-0                1/1     Running     0          8m27s
mimir-store-gateway-zone-b-0                1/1     Running     0          8m26s
mimir-store-gateway-zone-a-0                1/1     Running     0          8m27s
mimir-ingester-zone-c-0                     1/1     Running     0          8m27s
mimir-ingester-zone-b-0                     1/1     Running     0          8m27s
mimir-ingester-zone-a-0                     1/1     Running     0          8m26s
···

The same is true for the helm template command, some users use to deploy Mimir. The dashboards expect "mimir" template name:

% helm template grafana/mimir-distributed --name-template mimir

If you chose to use another release name, for the moment, you need to rebuild the dashboards yourself, following the guide.