Contact Point of AlertManager is absent

khuedoan / homelab

Fully automated homelab from empty disk to running services with a single command.

https://homelab.khuedoan.com

GNU General Public License v3.0

8.1k stars 742 forks source link

Contact Point of AlertManager is absent #133

Closed longbowapache closed 8 months ago

longbowapache commented 8 months ago

It seems that contact point of grafana is configured in values.yaml of grafana. However, the alert rules are configured in AlertManger and the alerts are not be notified. Is this a bug or intended design?

khuedoan commented 8 months ago

That's probably a bug, I'm still refactoring the alerting system. Could you please try sending an alert manually in Grafana (Alerting -> Contact points -> ntfy)?

longbowapache commented 8 months ago

That's probably a bug, I'm still refactoring the alerting system. Could you please try sending an alert manually in Grafana (Alerting -> Contact points -> ntfy)?

Sending an alert from grafana is ok now.

The problem is that the ntfy notification policy is belongs to Grafana, not AlertManager:

So, AlertManager's alert is delivered to null.

khuedoan commented 8 months ago

Ah right, thank you! I've been experimenting with both Grafana Alerting and Alertmanager, and I need to decide on using just one (my cluster is a bit messy right now to reproduce the issue - I'm planning to rebuild it in a few days though).

I haven't made a decision yet, but I'm now leaning towards yanking Grafana Alerting and only use Alertmanager (if so I'll need to replace the ntfy-relay from grafana-to-ntfy to something else for Alertmanager)

longbowapache commented 8 months ago

Thank you! and looking forward to your work!

khuedoan commented 8 months ago

There are some WIP if you're interested:

WIP generic webhook transformer https://github.com/khuedoan/webhook-transformer
WIP Transform Alertmanger to ntfy using the above project as a sidecar: https://github.com/khuedoan/homelab/compare/master...alertmanager2ntfy