search

ruby-shoryuken / shoryuken

A super efficient Amazon SQS thread based message processor for Ruby. This project is in MAINTENANCE MODE. Reach me out on Slack link on the description if you want to become a new maintainer.
Other
2.06k stars 280 forks source link

Preserve queue weights when unpausing #687

Closed cjlarose closed 2 years ago

cjlarose commented 3 years ago

644 introduced a change that unpauses FIFO queues when a message is processed from that queue. This prevents unnecessary delays, especially when using FIFO queues that contain messages with just a single unique message_group_id.

One side-effect of its implementation, however, reset queue weights in the WeightedRoundRobin polling strategy. In that class, @queues is an array of queue names, but some of the queue names are duplicated intentionally in order to represent weights. The change in #644 called uniq! on @queues, effectively resetting queue weights back down to 1 for all active (unpaused) queues. Since queue weights grow and shrink dynamically over time to adjust to the number of messages fetched from those queues, this is undesirable.

cc @davidrichey

davidrichey commented 3 years ago

@cjlarose I was able to take a spin with these updates locally making sure the changes unpause the queue as expected when a new message is on the queue at the time the message processes. Once this comment is confirmed/resolved I believe this will be good to go.