search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.37k stars 3.38k forks source link

The "production" Loki chart doesn't allow customizing the Alert PrometheusRules #11219

Open diranged opened 10 months ago

diranged commented 10 months ago

Describe the bug The production Loki chart's Alert Rules (https://github.com/grafana/loki/blob/helm-loki-5.36.3/production/helm/loki/src/alerts.yaml.tpl) are hard-coded with thresholds. These thresholds may not be appropriate for all environments, but they aren't customizable. If a user disables them (.Values.monitoring.rules.enabled=false) then the recording rules are also disabled which means that the dashboards all break.

To Reproduce Steps to reproduce the behavior:

  1. Install the current loki chart
  2. Set .Values.monitoring.rules.enabled=false
  3. See that dashboards are missing most graphs

Expected behavior Alerting thresholds should be customizable. Additionally the flags to enable Recording vs Alerting Rules should be broken out and separated.

Environment:

schahal commented 1 month ago

@JStickler I've tried to address this in https://github.com/grafana/loki/pull/13730. I'm not sure how to get a review/approval process started on that though.