search

sensu-plugins / sensu-plugins-hipchat

Handlers for Hipchat
http://sensu-plugins.io
MIT License
1 stars 16 forks source link

React appropriately to `HTTP 429` "rate limiting" response code #2

Open nhooey opened 8 years ago

nhooey commented 8 years ago

I'm getting an HTTP 429 from HipChat, which the documentation says is rate-limiting. It seems the Sensu plugin doesn't know about this response code:

{
  "timestamp": "2016-04-01T17:20:25.679807+0000",
  "level": "info",
  "message": "handler output",
  "handler": {
    "command": "handler-hipchat.rb",
    "type": "pipe",
    "name": "hipchat"
  },
  "output": [
    "/opt/sensu/embedded/lib/ruby/gems/2.2.0/gems/hipchat-1.5.1/lib/hipchat/room.rb:142:in `send': Unexpected 429 for room `2360342' (HipChat::UnknownResponseCode)
     from /opt/sensu/embedded/lib/ruby/gems/2.2.0/gems/sensu-plugins-hipchat-0.0.3/bin/handler-hipchat.rb:70:in `block in handle'
     from /opt/sensu/embedded/lib/ruby/2.2.0/timeout.rb:88:in `block in timeout'
     from /opt/sensu/embedded/lib/ruby/2.2.0/timeout.rb:32:in `block in catch'
     from /opt/sensu/embedded/lib/ruby/2.2.0/timeout.rb:32:in `catch'
     from /opt/sensu/embedded/lib/ruby/2.2.0/timeout.rb:32:in `catch'
     from /opt/sensu/embedded/lib/ruby/2.2.0/timeout.rb:103:in `timeout'
     from /opt/sensu/embedded/lib/ruby/2.2.0/timeout.rb:124:in `timeout'
     from /opt/sensu/embedded/lib/ruby/gems/2.2.0/gems/sensu-plugins-hipchat-0.0.3/bin/handler-hipchat.rb:68:in `handle'
     from /opt/sensu/embedded/lib/ruby/gems/2.2.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:55:in `block in <class:Handler>'"
  ]
}
majormoses commented 6 years ago

Basically we need to look at the response code and if its a 429 then we need a exponential backoff and retry strategy.

majormoses commented 6 years ago

I just realized that looking at the stack trace that is something that they should probably be handled by the upstream library. I need to do some research but this is actually a bug on their end from what I can tell so far.