search

ricbra / rabbitmq-cli-consumer

Consume RabbitMQ messages into any cli program
MIT License
247 stars 49 forks source link

Setting up to autoscale the consumer processes? #44

Closed andrefigueira closed 7 years ago

andrefigueira commented 7 years ago

Hi there,

I'm currently developing a solution for processing events based on this consumer where I work, however, we may want to scale the amount of workers based on the load, do you have any recommendation on how to achieve this?

Sorry if the wrong place to ask this, just wondering how you went about it?

EDIT: I'm using supervisord to manage the processes, but can only set numprocs as a hard limit

Kind Regards

ricbra commented 7 years ago

The most simple solution would be some kind of plugin for supervisord, which would scale when load gets high. But after some Googling around, I couldn't find anything useful. Another option would be modifying my consumer and make use of Go's excellent multithreading capabilities. I have no experience with that, so you're on your own if you follow that path.

andrefigueira commented 7 years ago

@ricbra Thanks for the reply, I'll take it under consideration, I think I'll try look for the plugin for supervisord, I think it sounds like the best place for this kind of thing. And thanks for this project all together!