search

ixti / sidekiq-throttled

Concurrency and rate-limit throttling for Sidekiq
MIT License
710 stars 76 forks source link

What happens when a job is throttled? #52

Closed jaredmoody closed 6 years ago

jaredmoody commented 6 years ago

After reading through the README there area few things I'm not clear on:

  1. What happens when a job is throttled? Do I have any control over what happens?
  2. Does concurrency only affect enqueued jobs or also scheduled jobs? Do I have any control over that?

Can someone who is familiar help me out?

Maybe these could be added to the README to improve the documentation?

ixti commented 6 years ago

1. What happens when a job is throttled?

It gets pushed back to the end of the queue it was retrieved from. And that queue is removed from the queues to poll for 2 seconds.

NB Throttling happens per-worker class (or shared strategy), but when throttling happens, queue where that job was pulled from is "paused" from polling for 2 seconds. That's a small sanity measure to avoid bombing redis with pull/push calls when queue contains only jobs that should all be throttled.

Do I have any control over what happens?

Not right now, but I'm open for proposals.

2. Does concurrency only affect enqueued jobs or also scheduled jobs?

TLDR; It affects both. Scheduled jobs are kept in a separate sorted set that sidekiq is polling on a regular basis and enqueues jobs that are ready to be executed (their time has come) into a normal queue they should be pushed to.

Do I have any control over that?

Not right now.

Maybe these could be added to the README

I'm not the best human-readable-texts writer. And most of this is purely technical details that uses should not care IMO - you don't need to know how does engine works in order to drive a car, only how to control it matters. :D

jaredmoody commented 6 years ago

Thanks for answering!

Just my two cents here but for me, knowing what actually happens when the job is throttled helped me determine if this would help solve my problem.

ixti commented 6 years ago

@jaredmoody Oh, I totally agree. I actually most of the time read sources than manuals as I'm not ready to use stuff I'm not sure how it works inside. ;)) So I totally understand your point, and now I actually agree that probably some tech details overview will be good to have in README or some sort of ARCHITECTURE-OVERVIEW. :D