[EPIC] Alerts Improvements

[ankitnayan]

• [ ] Allow multiple thresholds in a single alert and ability to route them to different notification channels
• [ ] Escalation capability
• [ ] AND conditions between queries. Eg, rate(otel-collector-exported-spans)=0 && rate(otel-collector-received-spans) > 10
• [ ] Chaining of alerts. Eg, Alert X and Alert Y both firing together
• [ ] Add 1 min and 2 min evaluation period. Maybe show caution when metrics is chosen asking to make sure scrape period is 30s
• [ ] https://github.com/SigNoz/signoz/issues/3727

Frontend

• [x] https://github.com/SigNoz/signoz/issues/4328
• [x] https://github.com/SigNoz/signoz/issues/3871
• [ ] https://github.com/SigNoz/signoz/issues/3622
• [x] https://github.com/SigNoz/signoz/issues/3733
• [ ] https://github.com/SigNoz/signoz/issues/3898
• [ ] https://github.com/SigNoz/signoz/issues/3734
• [ ] https://github.com/SigNoz/signoz/issues/4609
• [x] https://github.com/SigNoz/signoz.io/issues/1224

[pranay01]

Currently alerts in SigNoz are based on time series and a fixed threshold. Many use cases can be converted to this pattern. But the following use cases may not be easy to capture as a time series:

• Alert on events - Fire an alert when log contains a particular text or has a particular attribute
• Create alert when a time series crosses a dynamic band. e.g. If number of errors crosses more than 30% of the value in the last 5 minutes. Here there is no fixed threshold, but the threshold changes with time
• Change detection can only be done via clickhouse query not via query builder. e.g. Create alert if number of hosts sending metrics change by more than 10 in last 5 minutes. Currently, we don't have ability to compare across 2 time periods.
• Comparing to past time series e.g. Create an alert if the number of errors in logs every 5 minutes is more than 20% of what it was in the previous deployment
• No continuity alert - to detect if a certain event is happening continuously. Today this can be done via sum, but some just aggregates on the number of time an event has occured and not whether they have happened continuously or not