sensu-plugins / sensu-plugin

A framework for writing Sensu plugins & handlers with Ruby.
http://sensuapp.org
MIT License
126 stars 117 forks source link

No alert when critical events happen in silence period #182

Open jimzhou2010 opened 6 years ago

jimzhou2010 commented 6 years ago

Test process:

  1. Silence a check
  2. Then make the check from 0 (ok) to 2 (critical)
  3. After the silence expired, check still in critical status, but won't receive any alter unit refresh time reach.
  4. sensu-server log showing ""output":"only handling every 144 occurrences" (144 is refresh time/interval time)

Therefore, If in silence/stash period, any checks become critical, and after the time the silence expired, we still can't know the critical checks.

After silence expired, the critical occurrences is great than check occurrences setting, so that sensu won't trigger alert and just bail a "only handling every 144 occurrences".

Anyone can help checking on the issue ?

Appreciate for any help

majormoses commented 6 years ago

I feel like this needs to be addressed in the occurrences gem. ping @cwjohnston

majormoses commented 6 years ago

We might want another value to track so that we could say occurrences and occurrences_after_silence attributes so that we can handle that scenario better.