The Istio helm chart is routing /loki/api/v1/rules endpoint to the wrong container.

[p1-repo-sync-bot[bot]]

Bug

Description

Describe the problem, what were you doing when you noticed the bug?

The alerting page in Grafana is reporting that the Loki datasource is misconfigured. The istio chart is configuring the virtualservice in the logging namespace to route the /loki/api/v1/rules endpoint to the logging-loki-read container. The endpoint is on the logging-loki-backend container. Additionally, when modifying the virtualservice, the match rules for /loki/api/v1/push and /loki/api/v1/rules are both warned as "not used" because the /loki/api match rule precedes them.

BigBang Version

What version of BigBang were you running? 2.25.0

[p1-repo-sync-bot[bot]]

michaelmartin commented:

---

It seems from the Architecture mode here: https://grafana.com/docs/loki/latest/get-started/scalable-monolithic-mode.png , the alerts are only exposed in the backend pods -- which explains why we see this issue. Grafana expect the api and datasource url to be at the same host -- which in this case, it's not.

Upstream reported issue:

https://community.grafana.com/t/unable-to-fetch-alert-rules-is-the-loki-data-source-properly-configured/109766/8

https://github.com/grafana/grafana/issues/49618#issuecomment-1331424738

Still searching for a good solution.

[p1-repo-sync-bot[bot]]

jimmyungerman commented:

---

Moving issue into the Loki project to troubleshoot further.

[p1-repo-sync-bot[bot]]

Issue 'The Istio helm chart is routing /loki/api/v1/rules endpoint to the wrong container.' closed from GitLab side