search

bpaquet / node-logstash

Simple logstash implmentation in nodejs : file log collection, sent with zeromq
Other
517 stars 141 forks source link

High memory consumption if the target server is down #103

Closed tiagoalves closed 9 years ago

tiagoalves commented 9 years ago

We're using node-logstash (thanks for the great project, btw) to send data to a Graylog instance. If the Graylog server is down or a DNS error occurs, for instance, the node-logstash process's memory starts to grow until there's no more memory available.

It appears that this problem can be avoided using ZeroMQ and the back pressure settings. Is there a way to discard new messages or pause the log using the Gelf output?

Thank you.

bpaquet commented 9 years ago

Do you mean to implement the back pressure mechanism to Gelf Output plugin ?

On Mon, Apr 20, 2015 at 7:32 PM, Tiago Alves notifications@github.com wrote:

We're using node-logstash (thanks for the great project, btw) to send data to a Graylog instance. If the Graylog server is down or a DNS error occurs, for instance, the node-logstash process's memory starts to grow until there's no more memory available.

It appears that this problem can be avoided using ZeroMQ and the back pressure settings. Is there a way to discard new messages or pause the log using the Gelf output?

Thank you.

— Reply to this email directly or view it on GitHub https://github.com/bpaquet/node-logstash/issues/103.

tiagoalves commented 9 years ago

Yes, that's what I meant.

bpaquet commented 9 years ago

The problem is not detect a problem. It's about to detect the server is back online. This is handled in ZMQ at ZMQ level. With Gelf, the plugin can not know when the server is back on line. Solution :

Do you need to store messages in error to resend them when server is back on line ?

tiagoalves commented 9 years ago

I see. I agree with your first two points (have a way to monitor Graylog server state and an retry to send the messages every few seconds).

What I was expecting is that while Graylog is down, node-logstash would cache messages up to a certain configurable number or by cache size (in kb). If the maximum number of messages or cache size is reached, newer messages would simply be discarded. Personally, I could live with that but I don't know if that's what you expect and if it's the standard practice.

bpaquet commented 9 years ago

Which transport do you use to graylog ? Gelf with output gelf node-logstash plugin ?

tiagoalves commented 9 years ago

Yes, I use the Gelf output plugin: output://gelf://host:port

bpaquet commented 9 years ago

So, this plugin use UDP transport. Even if the destination is not reachable, there are no impact on node-logstash :)

What do you mean by dns error ?

On Mon, May 18, 2015 at 12:59 AM, Tiago Alves notifications@github.com wrote:

Yes, I use the Gelf output plugin: output://gelf://host:port

— Reply to this email directly or view it on GitHub https://github.com/bpaquet/node-logstash/issues/103#issuecomment-102868161 .