grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.93k stars 3.45k forks source link

Helm: default dashboards use a job label that doesn't exist #11390

Open alarobric opened 11 months ago

alarobric commented 11 months ago

Describe the bug The default Loki dashboards shipped with the Helm chart fail to load data, due to using a job label that doesn't match the metrics that are collected. The dashboard is querying metrics like this: loki_ingester_memory_chunks{cluster="$cluster", job=~"$namespace/(loki|enterprise-logs)-write"} but the ServiceMonitor is generating metrics with job=namespace/loki, not `job=namespace/loki-write'

To Reproduce Steps to reproduce the behavior:

  1. Installed loki in EKS using the Helm chart. Installed promtail, grafana, and kube-prometheus-stack alongside.
  2. Open the 'Loki/Chunks' dashboard for example. No data is shown.
  3. Drill into a chart widget, and modify the query to remove -write from the job label. Data shows as expected.

Expected behavior Default dashboards should work alongside the default ServiceMonitor configuration.

Environment:

Screenshots, Promtail config, or terminal output

alarobric commented 11 months ago

I guess this sounds like a related issue to #7657 or #9469