search

seomoz / qless-core

Core Lua Scripts for qless
MIT License
85 stars 33 forks source link

Throttling on arbitrary resources. #46

Closed james-lawrence closed 11 months ago

james-lawrence commented 10 years ago

Implements throttling for arbitrary resources. https://github.com/seomoz/qless-core/issues/27

Currently running in our production system with 14k worker processes spread across 15 redis processes.

Created a throttle object model in qless-core which consists of a active job set, a pending job set, and a maximum value. The throttle controls how many jobs that can run simultaneously.

Modified the put command to attempt to throttle new jobs. This is to prevent wasting pops to move jobs into the throttled state.

Modified the pop command to check if all the throttles for the job are available and if not to add it to pending queue for one of the throttles.

Modified the complete, fail, retry commands to release all throttles associated with the job.

When a throttle is released the throttle checks its pending job set and inserts up to (maximum - active) jobs into the waiting queue to be popped.

Let me know of any questions, changes that need to be made etc.

james-lawrence commented 10 years ago

Who can I reach out to about getting this merged in?

nitper commented 10 years ago

I would love to see this merged

dlecocq commented 10 years ago

Provisionally, this looks good. Since it's such a big change, I'm going to go over it a couple more times to give it the attention it merits.

james-lawrence commented 10 years ago

sounds good. Any questions you might have feel free to ask. Happy to have more eyes on it as you can see we found a bug as recently as a few days ago.

james-lawrence commented 10 years ago

I'll remerge mainline into this branch in a week or so hopefully.

james-lawrence commented 10 years ago

though this will likely be the last update I do as the two mainlines continue to diverge and without this chunk of work its not worth rectifying the differences on our end.