Check flap detection

[portertech]

Sensu 1.X provides check flap detection, using the same algorithm as Nagios. Sensu stores the 21 latest exit statuses for every check and uses them to calculate a weighted "total state change %" value, with the more recent exit statuses having more value. The "total state change %" is then compared against two configurable thresholds, the high threshold used to determine if a check enters a flapping state, and the low threshold use to determine if a check exits a flapping state.

The limited Sensu 1.X check flap detection documentation can be found at: https://sensuapp.org/docs/latest/reference/checks.html#check-attributes under "low_flap_threshold" and "high_flap_threshold".

The Sensu 1.X "total state change %" calculation Ruby code and YARDOC can be found at: https://github.com/sensu/sensu/blob/b27e506875e466f2192e00c4da8dfd6344819309/lib/sensu/server/process.rb#L380-L415

The Sensu 1.X flap detection Ruby code and YARDOC can be found at: https://github.com/sensu/sensu/blob/b27e506875e466f2192e00c4da8dfd6344819309/lib/sensu/server/process.rb#L417-L453

Sensu 2.X could implement the total state change % calculation and flap detection in eventd.

[grepory]

• Add configuration for low flap threshold
• Add configuration for high flap threshold
  • Considered flapping if above high watermark
• Add nagios-algorithm to eventd
• Event as flapping

Further work: figure out a way to better represent flapping checks, but that is out of scope for this ticket.

[grepory]

Make sure that we have at least as much test coverage as 1.x.

[portertech]

The 1.X spec https://github.com/sensu/sensu/blob/master/spec/server/process_spec.rb#L206-L253

[palourde]

CoS

• [x] All 1.x test cases pass

Tasks

• [x] Implement 1.x test cases
• [x] Add configuration for low flap threshold
• [x] Add configuration for high flap threshold
• [x] Add nagios-algorithm to eventd

[portertech]