search

ruby-shoryuken / shoryuken

A super efficient Amazon SQS thread based message processor for Ruby. This project is in MAINTENANCE MODE. Reach me out on Slack link on the description if you want to become a new maintainer.
Other
2.06k stars 280 forks source link

Question: how to optimally configure priority queues with long polling #695

Closed bforma closed 2 years ago

bforma commented 2 years ago

Hi,

We have two separate SQS queues, one for 'default priority' messages and one for 'low priority' messages.

I've tried to configure handling these with Shoryuken as follows:

polling_strategy: StrictPriority
queues:
  - default_priority
  - low_priority

But when we use this with long polling (Shoryuken.sqs_client_receive_message_opts = {wait_time_seconds: 20}) the throughput drastically decreases (because each queue is checked one by one with a wait time of 20 seconds in between).

As a workaround we now run most Shoryuken workers with -q default_priority and several with -q low_priority.

Any thoughts on a more optimal configuration/setup?

github-actions[bot] commented 2 years ago

This issue is now marked as stale because it hasn't seen activity for a while. Add a comment or it will be closed soon.

github-actions[bot] commented 2 years ago

This issue was closed because it hasn't seen activity for a while.