search

bpaquet / node-logstash

Simple logstash implmentation in nodejs : file log collection, sent with zeromq
Other
517 stars 141 forks source link

AMQP can not listen to queue? #106

Closed janober closed 8 years ago

janober commented 9 years ago

I am new to logstash & RabbitMQ. So maybe I understand it totally wrong but I do not understand why I can only listen to an exchange and not to an already existing queue. Isn`t one of the biggest advantages of having RabbitMQ in front of logstash that it has a queue. So that my servers can still send messages to an exchange which got bound to a permanent queue even if logstash is currently down? So as soon as it comes up again it can consume the messages which would normally have been lost?

janober commented 9 years ago

Just saw that to normal logstash has that option ("queue"): https://www.elastic.co/guide/en/logstash/current/plugins-inputs-rabbitmq.html

Guess I gonna have to use that version after all for now.

bpaquet commented 9 years ago

Queue is not implemented because no one asked for it :)

By the way, using queue seems a little bit strange to me : logs are ephemeral and not critical data, the transport you use can lost some messages. For example, the standard transport, syslog, use udp. So if you need reliability to transport your log messages, and a queue semantic (only one consumer can consume a message), a queue in RabbitMQ can be a solution. But beware of performance : if your consumer is not present, your RabbitMQ can be overloaded of logs messages.

Anyway, the standard utilization of RabbitMQ for logs is simple exchange (if no consumer is present, messages are lost), or topic routing using log type.

You can submit a PR with queue implementation, I think only few lines of code are needed.

On Fri, Jun 5, 2015 at 7:37 PM, Jan notifications@github.com wrote:

Just saw that to normal logstash has that option ("queue"):

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-rabbitmq.html

Guess I gonna have to use that version after all for now.

— Reply to this email directly or view it on GitHub https://github.com/bpaquet/node-logstash/issues/106#issuecomment-109372811 .

janober commented 9 years ago

Ah yes that makes sense. Yes the overload could probably be an issue if I would log a lot and they would not get consumed for a long time. Should probably do some testing to see how many messages it can take before that happens.

Will then have a look because of the queue implementation.

janober commented 9 years ago

You were right. Was quite easy. Did submit the pull request.