Sneakers concurrency from RabbitMQ consumer perspective

[pradosh987]

Sneaker seems to be excellent gem for RabbitMQ but I am having trouble grasping its abstraction written over RabbitMQ consumer. In RabbitMQ you can create as many consumer with set amount of prefetch. I have done this in Nodejs where I'll create certain number of consumers on certain machines. Same things appear to be little complicated in rails with workers and threads and worker classes. From what I understand, each worker class will have its thread pool, say 10 as configured globally. Say I have 20 worker classes and 4 number of per-cpu worker configured in Sneaker configuration.

So

1. how many consumers will be running at any point of time?
2. Will there be 20 classes* 10 threads per class = 200 threads?
3. What does 4 worker per-cpu process represent?

Can anyone please elaborate on this? Thanks in advance.

[pradosh987]

Anyone?

[thisisDom]

@pradosh987 From my understanding,

1. The number of consumers would be 4 workers * 20 classes = 80 consumers.
2. Correct. The number of messages that can be concurrently consumed is the sum of all the class threads, so 200 messages in your example case.
3. It's a bit confusing when it says "per-cpu" in the documentation. The number of workers represents the number of processes that will be forked by the operating system, allowing for messages to be run in parallel. In your example. 4 processes will be created, each with their own set of 80 consumers and a total of 200 threads. Now how many of these actually run in parallel depends on the hardware running RabbitMQ.

Please, correct me if I'm wrong.

[pradosh987]

@pradosh987 From my understanding,

1. The number of consumers would be 4 workers * 20 classes = 80 consumers.
2. Correct. The number of messages that can be concurrently consumed is the sum of all the class threads, so 200 messages in your example case.
3. It's a bit confusing when it says "per-cpu" in the documentation. The number of workers represents the number of processes that will be forked by the operating system, allowing for messages to be run in parallel. In your example. 4 processes will be created, each with their own set of 80 consumers and a total of 200 threads. Now how many of these actually run in parallel depends on the hardware running RabbitMQ.

Please, correct me if I'm wrong.

Thanks for nice explanation @thisisDom that helps, sorry I moved on from ruby to node and didn't came back here.

1600 consumers??? That's totally getting out of hand. How many rabbitmq connections it will create?

[michaelklishin]

It's worth mentioning that Sneakers uses Bunny, a RabbitMQ client, under the hood, which uses a thread pool per channel for delivery dispatch. By default the pool size is 1 to avoid concurrent delivery processing which may or may not be expected.

[lucasintel]

• One connection is open per process (runner workers).
• Each worker class (classes that includes Sneakers::Worker) will have a thread pool of size threads for processing incoming messages. One channel is open.
• If share_threads is true, a single thread pool will be shared across all workers in the same process.

The design is really fine.