search

github-education-resources / classroom

GitHub Classroom automates repository creation and access control, making it easy for teachers to distribute starter code and collect assignments on GitHub.
https://classroom.github.com
1.35k stars 569 forks source link

Improve Sidekiq queue efficiency #1381

Open BenEmdon opened 6 years ago

BenEmdon commented 6 years ago

Feature request :sparkles:

Currently classroom uses sidekiq queues inefficiently. This is our current queue setup: https://github.com/education/classroom/blob/4f9db7e412c57ce8aeb328a355b223e0a7254546/config/sidekiq.yml#L7-L14

This is what the Sidekiq docs say about having lots of queues:

I don't recommend having more than a handful of queues. Lots of queues makes for a more complex system and Sidekiq Pro cannot reliably handle multiple queues without polling. https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues

One way we could improve this would be to use the regular 3 queues (critical, default, and low).

stale[bot] commented 6 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.

v4vasanth commented 5 years ago

Hi! I can help with this ✋

gustavosobral commented 4 years ago

I think I can do this one. Have you done any work on this @v4vasanth?

@BenEmdon So, your idea here is to just create these three queues that you've mentioned and re-assign the workers? I thought about doing something like this:

:queues: # https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues
  - [critical, 3]
  - [default, 2]
  - [low, 1]

And assigning the Workers that have a queue with priority 3 to the critical queue and so on.

Other option would be this setup:

:queues: # https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues
  - critical
  - default
  - low

But from the Sidekiq documentation we have that:

This means that any job in the default queue will be processed only when the critical queue is empty.

I don't like the idea of only processing from default when critical is totally empty. So I will go for the option one.