search

Yelp / sensu_handlers

Custom Sensu Handlers to support a multi-tenant environment, allowing checks themselves to emit the type of handler behavior they need in the event json
Apache License 2.0
75 stars 31 forks source link

Make the pagerduty handler create events whenever sensu creates events. #49

Closed solarkennedy closed 9 years ago

solarkennedy commented 9 years ago

Primary: tdoran

We currently try to never send warnings to PD.

However, this causes some bugs/oddities because our filtering logic considers 1,2 to be creation events and 0 to be resolve, where as the PD handler consides them 2 to be creation events and 0,1 to be resolve events.

This causes some "resolve events" to never make it through, I think, and generally seems weird. It also causes unnecessary spam to the PD api as we spam it with "resolves" constantly whenever we have a warning going off.

I think we should give up on this and make our PD handler fit more in line with what sensu is actually doing. (making the create and resolve events align)

If we don't want warnings to go to PD, then we shouldn't be making monitoring checks that are set to page that have bogus warning levels that we don't actually care about?

solarkennedy commented 9 years ago

@bobtfish can you do a code review + comment?

jaxxstorm commented 9 years ago

This looks good to me btw :thumbsup:

bobtfish commented 9 years ago

Shipit, but this is also likely to be noisy / disruptive in the short term I think?

Sensu keepalive goes from 180 > 120 seconds MEM_FREE will start alerting more (5% > 10%) crossdc_www_high threshholds need adjusting

(from a quick glance at the last few days irc logs)