Grafana's /alerting/list doesn't show any alerts via datasource plugin

[chouzee]

Hi! I try to list alerts in Grafana using the plugin as a datasource but nothing shows up but they are show when I don't use the plugin. The url for both datasources are the same. Could you clarify if the plugin supports this feature?

[chouzee]

Forgot to mention: for both datasource I turned on Manage alerts via Alerting UI

[dmitryk-dk]

Hi @chouzee ! I have checked your question.

1. I created a test alert

   

2. Went to alerting tab

   

3. I Checked the List tab, and I had no error here

[dmitryk-dk]

@chouzee can you share what version you use? Your error said that something was wrong with the network.

[yuriydzobak]

@dmitryk-dk this one feature https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1739 and https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2583 like, read rules, alerts from VMalert. It worked in datasource prometheus

[dmitryk-dk]

@yuriydzobak ! Thank you, I will look at those issues and test them.

[hagen1778]

@dmitryk-dk I can confirm alerting rules aren't working with VictoriaMetrics DS plugin. It needs to be fixed. It is likely, plugin doesn't forward alerts request to the VictoriaMetrics backend.

Steps to reproduce:

1. Setup Grafana with VictoriaMetrics plugin
2. Spin up VictoriaMetrics single node ./bin/victoria-metrics --vmalert.proxyURL=http://localhost:8880
3. Spin up vmalert with some alerting rules ./bin/vmalert -rule=/alerting/rules -datasource.url=http://localhost:8428 -notifier.url=http://localhost:9093
4. Open Grafana and go to Alerting tab

Expected result:

Actual result:

Please note, both datasources are pointing to the same URL

[dmitryk-dk]

Hi @hagen1778 , @yuriydzobak , @chouzee ! I have found the difference between grafana plugin and our implementation. I think we have a possible solution and @Loori-R will try to do it on FE part

[dmitryk-dk]

Hi @hagen1778 , @yuriydzobak , @chouzee ! I have investigated the issue and created two issues in the Grafana repository. The main problem is when data source type differs from prometheus or loki grafana only calls internal API for alerts created from the Grafana UI. There are two issues with details explanations. https://github.com/grafana/grafana/issues/67466 https://github.com/grafana/grafana/discussions/68169

[hagen1778]

So it is like Grafana doesn't make the API call to fetch the alerts if your datasource isn't prometheus or loki? Is my understanding correct?

[dmitryk-dk]

So it is like Grafana doesn't make the API call to fetch the alerts if your datasource isn't prometheus or loki? Is my understanding correct?

They make API call, but they use the default source name grafana when to build URL to fetch

`/api/prometheus/${getDatasourceAPIUid(dataSourceName)}/api/v1/rules`,

function getDatasourceAPIUid(dataSourceName) returns grafana instead of the datasource_ID.

But even if we do some tricks with Grafana code the backend code has some conditions which prevent the request to data source different from Prometheus or loki. When I remove those conditions and add datasource type like victoriametrics to the places where grafana checks datasource type it works as expected.

[vainkop]

That seems like a fix for the issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1739#issuecomment-1118394988

  -vmalert.proxyURL string
     Optional URL for proxying alerting API requests from Grafana. For example, if -vmalert.proxyURL is set to http://vmalert:8880 , then requests to /api/v1/rules are proxied to http://vmalert:8880/api/v1/rules

[dmitryk-dk]

vmalert.proxyURL Hi @vainkop ! I don't think this will help, because Grafana itself determines where to send the request. The problem is that Grafana stores encoded data that supports aletings

[vainkop]

vmalert.proxyURL Hi @vainkop ! I don't think this will help, because Grafana itself determines where to send the request. The problem is that Grafana stores encoded data that supports aletings

So at the moment the Grafana (6.59.*) used in https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-k8s-stack can be considered as not full compatible with VMalert (v1.94.0) from the same chart? :(