search

enova / sidekiq-rate-limiter

Redis-backed, per-worker rate limits for job processing.
http://rubygems.org/gems/sidekiq-rate-limiter
MIT License
193 stars 43 forks source link

Rate limiter "slows down" over time under sustained workload #10

Closed ryana closed 9 years ago

ryana commented 9 years ago

Hey guys,

I noticed that the rate limiter can "slow down" after running for a long time:

Sidekiq performance

You can see it was really spread out until I did this in a Ruby console: irb(main):070:0> Sidekiq.redis {|c| c.del("sidekiq-rate-limit:FcWorker") }

The moment I did that, you see how the spikes became much more localized.

It's like the rate limiter "leaks" some how. This is a sidekiq worker running on Heroku. I'll continue to monitor the situation, but it's only been running for ~ 6 hours, so I'm expecting it to pop up again several times over the next couple days.

In this case, I have 500k queued with a 600/minute rate limit. Is this a known bug when you have a constant stream of jobs?

Oh and as I'm writing this, you can already see how it's spreading out over time: Sidekiq performance #2

ryana commented 9 years ago

You can also see how the "humps" start to take the same shape:

Humpkiq

I think the issue is the rounding of Time in redis_rate_limiter. It does a Time.now.to_i, which truncates the decimal. This effectively "adds" time along the way. This commit changes the shape to this:

Spikekiq

So it turns out this is a redis_rate_limiter issue. I'll move this ticket over there.

bwthomas commented 9 years ago

Wow, seriously thanks for the diligence.

ryana commented 9 years ago

:-D