Shoryuken config

[jrhoads]

Purpose

Current shoryuken workers sit idle waiting for work.

We were using groups and concurrency limits as a proxy for priority/weighted queue.

There are other mechanisms for that. https://github.com/ruby-shoryuken/shoryuken/wiki/Polling-strategies

closes: Add github issue that originated this PR

Approach

Changes the concurrency configuration for the shroyuken workers.

• The default polling strategy is weighted round robin
• Drops groups and individual concurrency limits and adds weights to the queues.
• Adds option to set total concurrency for all queues as an environment variable SHORYUKEN_CONCURRENCY.
• Default SHORYUKEN_CONCURRENCY is 15 (the same as the previous concurrency limit).

Learning

Using the shoryuken docs as a reference. Specifically, the load balancing, concurrency and polling strategies sections

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)

• [x] New feature (non-breaking change which adds functionality)

• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Reviewer, please remember our guidelines:

• Be humble in the language and feedback you give, ask don't tell.
• Consider using positive language as opposed to neutral when offering feedback. This is to avoid the negative bias that can occur with neutral language appearing negative.
• Offer suggestions on how to improve code e.g. simplification or expanding clarity.
• Ensure you give reasons for the changes you are proposing.

[jrhoads]

Current dependencies on/for this PR:

• master
  • PR #1092 👈

This stack of pull requests is managed by Graphite.

[jrhoads]

I think you are correct. The groups structure overwrote (within the group) the default configurations set at the top. Will set the fallback SHORYUKEN_CONCURRENCY to 30