search

grafana / grafana

The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
https://grafana.com
GNU Affero General Public License v3.0
65.29k stars 12.18k forks source link

Alerting: transient "Failed to load rules config from Loki: Ruler API is not available for Loki" error #80895

Open katya-titkova-oura opened 10 months ago

katya-titkova-oura commented 10 months ago

What happened?

When I open an existing Grafana-managed alert rule or try to create a new one in the UI, and then close either of them without saving (by clicking "Alert rules"), I get "Failed to load rules config from Loki: Ruler API is not available for Loki" error. As soon as I refresh the page, the error is gone.

Ruler API is enabled and Ruler evaluates the alerting rules that are stored locally. Those alerting rules are also visible by Grafana.

What did you expect to happen?

No error all the time.

Did this work before?

It didn't work before, but the error is also appearing only about 70% of the time when I follow the steps below.

How do we reproduce it?

  1. Go to Alerting -> Alert rules

  2. Click "New alert rule"

  3. Change data source to Loki

  4. Click "Alert rules" (either above or at the sidebar)

    Screenshot 2024-01-19 at 15 23 37

  5. The error appears

    Screenshot 2024-01-19 at 15 24 31

  6. Refresh the page

  7. The error is gone

    Screenshot 2024-01-19 at 15 24 43

Is the bug inside a dashboard panel?

No response

Environment (with versions)?

Grafana: 10.1.6 OS: macOS Sonoma 14.2.1 Browser: Chrome 120.0.6099.234

Grafana platform?

Kubernetes

Datasource(s)?

Loki

katya-titkova-oura commented 10 months ago

I switched from local storage to s3 and can't reproduce this anymore.

tonypowa commented 9 months ago

Hi @katya-titkova-oura Thank you for reporting this issue I am not sure that this issue is related https://github.com/grafana/grafana/issues/49618 but still, have a look. In the meantime, I will send this to the maintainers for review

tonypowa commented 9 months ago

cc @grafana/alerting-squad @grafana/observability-logs

ivanahuckova commented 9 months ago

This issue is for @grafana/alerting-squad and shouldn't be related to Loki data source. So moving it from our board and keeping it in alerting.

Ayatallah commented 9 months ago

I switched from local storage to s3 and can't reproduce this anymore.

How did you manage to sync Loki Alert rules in S3 bucket?

gillesdemey commented 9 months ago

Hi! Loki does not enable the alert rules ruler API required for Grafana Alerting when using local storage.

From the Loki docs;

The local implementation reads the rule files off of the local filesystem. This is a read-only backend that does not support the creation and deletion of rules through the Ruler API.

source: https://grafana.com/docs/loki/latest/alert/#ruler-storage

armandgrillet commented 9 months ago

Gilles' comment is right but we should display this warning all the time, and not periodically. This is a front-end issue.