search

SigNoz / signoz

SigNoz is an open-source observability platform native to OpenTelemetry with logs, traces and metrics in a single application. An open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool
https://signoz.io
Other
18.87k stars 1.23k forks source link

Bug in test notifications #5459

Open makeavish opened 3 months ago

makeavish commented 3 months ago

After 1st test notification, subsequent test notifications are not fired until previous test notification is resolved or it's been 5 mins since it was fired. https://www.loom.com/share/fa6c6e9ac05848d9ae3b066aa212f308?sid=f3704a6e-5461-4f1a-8fc2-e13fe184bd22

<@U037WFAUC69>

Slack Message

Ellipse0934 commented 3 months ago

Simple fix is to add an arbitrary label or change the alert name. This will lead to alert.Fingerprint to change in alertmanager.

We can add it in pkg/query/service/rules/manager.go

-   parsedRule.AlertName = fmt.Sprintf("%s%s", alertname, TestAlertPostFix)
+       parsedRule.AlertName = fmt.Sprintf("%s%s", alertname, TestAlertPostFix + "_" + randstring(5))

Or drop TestAlertPostFix from the backend and have this logic in the frontend. Or add a single random label in the frontend.

Ellipse0934 commented 3 months ago

If you want me to file a PR, let me know @makeavish .