search

ecotoneframework / ecotone-dev

Ecotone Framework Development - This is Monorepo which contains all official public modules
https://docs.ecotone.tech
Other
37 stars 16 forks source link

Support consuming multiple queues with a single command #357

Closed benr77 closed 2 months ago

benr77 commented 2 months ago

Currently to consume messages from a queue you run a command, for example in Symfony:

bin/console ecotone:run async

It would be very useful to be able to consume multiple queues using a single worker. For example:

bin/console ecotone:run queue1 queue1 queue2
bin/console ecotone:run --all

For my use case, I have several really small apps running on Digital Ocean app platform for $5/month, but it uses two different queues (one for local async processing, the second a distributed bus).

To add these workers it's an additional $5/month which is tripling the cost for each app. If it were under huge load this obviously would not be a problem, but for small low-traffic apps having a single worker for all queues would be really helpful.

Thanks

lifinsky commented 2 months ago

You can use multiple Docker containers or multiple supervisor-managed processes and have multiple consumer processes on a single instance (DigitalOcean Droplet). All consumers should be independent.

benr77 commented 2 months ago

This is exactly what I would do if I were running the app on a DO Droplet.

However, DO App Platform is their serverless offering and here we boot a single Docker container per "app", hence running into the issue I describe.

lifinsky commented 2 months ago

You can run one single process - supervisor for multiple consumers management. http://supervisord.org/subprocess.html