search

Yelp / sensu_handlers

Custom Sensu Handlers to support a multi-tenant environment, allowing checks themselves to emit the type of handler behavior they need in the event json
Apache License 2.0
75 stars 31 forks source link

add rate limiting to jira handler #90

Closed somic closed 8 years ago

somic commented 8 years ago

Very experimental, please don't merge - if we agree in principle something like this is desired, I will do more testing and will merge.

I would like to make our handlers smarter. Thought adding simple rate limiting to jira handler could be a good place to start.

Main idea is if N sensu clients trigger event foo, when rate limiting is requested (via monitoring_check sensu_custom field where the limit will be set), we will only create up to certain number of jira issues per minute.

This will have some tricky interactions with our alert_after logic and filtering so actual effect of this change may not be as clear cut but the idea is to avoid spamming jira.

A check must opt in to this and handler should be fully backwards compatible.

Thoughts? Again - please don't merge yet.

solarkennedy commented 8 years ago

Seems fine to me? This is just to solve ticket floods right?

somic commented 8 years ago

Correct, immediate goal is to be able to solve ticket floods. I do want to continue building out context awareness and potentially more state in our handlers, in order to be able to better support complexity of our environments.

solarkennedy commented 8 years ago

Steal the gemfile from here: https://github.com/Yelp/puppet-uchiwa/blob/master/Gemfile

I recently went on a spree of pinning gemfiles, similar to what you are trying to do here, and missed this module. (needs to be merged carefully because sensu gems and stuff.

somic commented 8 years ago

@solarkennedy Just saw your message, after travis is already green :) One of these days we will drop ruby 1.8 and simplify all our gemfiles...