search

logstash-plugins / logstash-integration-rabbitmq

Logstash Integration Plugin for RabbitMQ, including Logstash Input and Output Plugins
Apache License 2.0
5 stars 25 forks source link

Multiple consumers to support rabbitmq-sharding #20

Open michaelklishin opened 9 years ago

michaelklishin commented 9 years ago

rabbitmq-sharding is an opinionated plugin which uses multiple "physical" queues behind one logical queue, taking advantage of more CPU cores.

This sounds like a great fit for the Logstash use case. However, to support it, the input needs to use multiple consumers.

This can be an option or an alternative input. Would there be any interest in this?

andrewvc commented 8 years ago

@michaelklishin I like this idea. Would you be up to contribute a patch here?

michaelklishin commented 8 years ago

@andrewvc yes, in a few weeks.

GlenRSmith commented 8 years ago

I'm quite interested in seeing improvements in throughput where RabbitMQ and Logstash are both in the data pipeline. (Evidence here: https://github.com/logstash-plugins/logstash-output-rabbitmq/issues/25)

Is the proposal here to add configuration options to the ls-in-rmq plugin which would only work against an exchange that has the linked plugin installed?

andrewvc commented 8 years ago

@michaelklishin thinking through this I think that this should be a separate plugin if it's tightly bound to rabbitmq sharding. That being said if it's as simple as patching this to support multiple input queues (so you could pass an array of queues as an argument) then that would be a good patch to take.

lsdev14 commented 5 years ago

Hey @michaelklishin, @andrewvc and @GlenRSmith any news about this topic? Has any task started about it?

andrewvc commented 5 years ago

@lsdev14 I no longer work on logstash, but @jsvd may be able to help here.