Can concurrency be shared across groups (Managers) ?

[gurukannappa]

Lets say there are 5 processing groups and a concurrency value of 10, which means each group gets a concurrency of 10 and a total of 50 threads will be created. If there are groups that only occasionally receive messages then all of the configured concurrency sit idle and if there are other busy groups that needs extra threads it would limit its processing capabilities. How about if there is an option to maintain this concurrency pool globally where groups can share the concurrency when they are not in use.

concurrency: 10
groups:
  group1:
    concurrency: 10
    queues:
      - queue1
  group2:
    concurrency: 10
    delay: 10
    queues:
      - queue2
     .
     .
     .
  group5:
    concurrency: 10
    delay: 10
    queues:
      - queue2

[phstc]

If you want to share concurrency globally, you could remove the groups.

concurrency: 25
queues:
  - queue1
  - queue2
  - queue3

The configuration above will provision 25 threads shared with three queues.

There's also a load balancing option that can be an option depending on your need.

[gurukannappa]

@phstc yes this global option with group removal has one limitation when we enable long polling. If a queue in the middle and has a long polling of 20secs then rest of the queues after a long polling queue would be waiting. as you have mentioned it in this issue https://github.com/ruby-shoryuken/shoryuken/issues/30#issuecomment-70343041 . For groups its safe to enable long polling as the long polling does not block other groups, In this case imagine each group needs to have its own concurrency. Where as in the global queue model the concurrency is shared by all queues in the list. What am thinking is can we mix the group model with the global queue concurrency model ? so that groups can process independent of the long polling and concurrency can be shared effectively between groups. Please let me know your thoughts on this.

[phstc]

Suppose you have 10 queues and all of them receive 10 messages at the same time. You will need 100 processors concurrency=100 to process all of them.

This is not recommended because of the dead letters queue as it will increase the receive count.

@gurukannappa how would that prevent the issues discussed in the thread you linked?

[gurukannappa]

Ahh, the comment that i linked is a reference to queue blocking with long polling enabled. Could you help me if there is a something that can fit the following spec?

1. There are around 30 sqs queues in the application and every queue needs to have long polling enabled to reduce cost.
2. If long polling is enabled one queue should not be blocking other queues.
3. Application has a concurrency limit of 10 per process. Increasing the concurrency chokes the database connections and hence the worker operates around a concurrency of 10. but concurrency of 10 per queue would be an overkill. instead a concurrency of 10 should be shared by all queues but in groups.

[phstc]

$1.30 is like 2.600,000 requests and a regular month (30 days) has 2.592,000 seconds. Not counting that the first 1M requests is FREE. https://github.com/ruby-shoryuken/shoryuken/issues/30#issuecomment-70344893

needs to have long polling enabled to reduce cost.

@gurukannappa is the empty poll cost worth it compared to the complexity?

the item gets pushed back on the queue and gets processed next time.

This is not recommended because of the dead letters queue as it will increase the receive count. https://github.com/ruby-shoryuken/shoryuken/issues/30#issuecomment-70343459

We can't use threads while waiting for the long poll. Otherwise, if all queues return messages, Shoryuken won't have enough threads to consume them, and that can cause issues with visibility timeout, receive count and dead letter queues.

[gurukannappa]

@phstc thanks for providing this insight. Will take up the recommendations from here. :)

[phstc]

@gurukannappa happy to help! The biggest issue is needing threads available when the long poll returns messages.