search

sensu / sensu-go-fatigue-check-filter

An event filter for Sensu Go for managing alert fatigue
MIT License
6 stars 8 forks source link

Alert on occurrences only when within a time window #27

Open raags opened 3 years ago

raags commented 3 years ago

I have a use case of alerting on occurrences, but only if they happen within a time window.

For e.g. alert if there are 3 consecutive events in 1 hour.

This need arises especially when generating alerts via the event API, and not via regular checks. So, let's say you have a cronjob that is known to fail intermittently, and we have set occurrences to 3. Without the time window, an alert will be generated every time occurrences cross 3, even if they happen over a week.

nixwiz commented 3 years ago

That is not a use case for this particular filter as its only time based setting is a based on allowing failures to be passed on the handler(s) based on intervals of occurrence. There is no configuration available for it to say x events within y time frame.

If your occurrences for this can reach 3 over the course of a week, but this cron job is seemingly running multiple times per hour (based on your wanting the third occurrence within an hour), it sounds like you are not sending events with status = 0 for successful completions of the cronjob. Are you sending successful status events?

raags commented 3 years ago

Yeah, cron job wasn't a good example. If success can be sent, then I suppose we don't need this. But there are still cases where success cannot be sent. For e.g. application sending alerts due to a failure, where sending success would be too frequent (for e.g a web app).

It's actually doable by using the check history. That has the state for all past occurrences with the time when they occurred. I'll take a stab at it.