The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
We use our in-house service that act as alert manager. We have added this service as one of the alertmanager datasource where all grafana managed alerts, Mimir & Loki alert are sent. This works perfectly fine till we enabled SSL on our alertmanager service.
After enabling SSL on the alert manager service we updated external alertmanager config, mimir ruler alert manager config & loki ruler alertmanager config to use the certificate & key while communicating with alertmanager service.
The Configuration looks like below
logger=ngalert.sender.external-alertmanager t=2023-08-23T13:03:07.184666046Z level=error alertmanager=https://\.\:\/api/v2/alerts count=1 msg="Error sending alert" err="Post \"https://\.\:\/api/v2/alerts\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
What did you expect to happen?
Alertmanager should consider tlsSkipVerify flag and should not validate server certificate. If this tlsSkipVerify is working fine as expected then I am not sure why the same configuration works for Mimir & Loki Ruler , but not for Grafana Alert manager
Did this work before?
Yes the setup works fine if we remove certificates from everywhere (Grafana alert manager, Mimir Ruler, Loki Ruler & Alertmanager service)
How do we reproduce it?
Add a self signed certificate to the alert manager service
Configure client key & certificate in Grafana alertmanager setting
What happened?
We use our in-house service that act as alert manager. We have added this service as one of the alertmanager datasource where all grafana managed alerts, Mimir & Loki alert are sent. This works perfectly fine till we enabled SSL on our alertmanager service. After enabling SSL on the alert manager service we updated
external alertmanager config
,mimir ruler alert manager config
&loki ruler alertmanager config
to use the certificate & key while communicating with alertmanager service. The Configuration looks like belowMimir & Loki Ruler Configuration --- Works
Grafana External Alertmanager configuration --- Does not Work
ERROR
The error that I see in the grafana logs
What did you expect to happen?
Alertmanager should consider
tlsSkipVerify
flag and should not validate server certificate. If thistlsSkipVerify
is working fine as expected then I am not sure why the same configuration works for Mimir & Loki Ruler , but not for Grafana Alert managerDid this work before?
Yes the setup works fine if we remove certificates from everywhere (Grafana alert manager, Mimir Ruler, Loki Ruler & Alertmanager service)
How do we reproduce it?
Is the bug inside a dashboard panel?
No response
Environment (with versions)?
Grafana: 10.0.3
Grafana platform?
Kubernetes
Datasource(s)?
No response