search

bpaquet / node-logstash

Simple logstash implmentation in nodejs : file log collection, sent with zeromq
Other
517 stars 141 forks source link

Please separate emitted logs with a newline to work around logstash bug #52

Closed davux closed 10 years ago

davux commented 10 years ago

I run node-logstash with the following parameters: $ ./node-logstash-agent input://file:///tmp/foo.log output://tcp://host.example.com:5001

On the receiving side, logstash (not node-logstash) configured with "tcp" input cannot parse that content. The reason is that currently logstash can only parse newline-separated logs, because of the JSON-parsing library it uses (no Ruby library is capable of parsing JSON as a stream).

Here is a sample of what node-logstash sends over:

 {"message":"xxx","host":"hostA","@timestamp":"2013-12-19T00:09:03.995Z","@version":"1"}{"message":"yyy","path":"/tmp/logbench/1.log","host":"hostA","@timestamp":"2013-12-19T00:09:03.995Z","@version":"1"}{"message":"zzz", ... }

Strictly speaking, this is a logstash bug. However, it's not likely to be fixed soon, so in the meantime it would be nice to work around it by inserting a newline between traces, which doesn't introduce any problem anyway since it's JSON.

bpaquet commented 10 years ago

Just add ?serializer=raw in your output config.

output://tcp://host.example.com:5001?serializer=raw

On Thu, Dec 19, 2013 at 3:10 AM, davux notifications@github.com wrote:

I run node-logstash with the following parameters: $ ./node-logstash-agent input://file:///tmp/foo.log output://tcp:// host.example.com:5001

On the receiving side, logstash (not node-logstash) configured with "tcp" input cannot parse that content. The reason is that currently logstash can only parse newline-separated logs, because of the JSON-parsing library it uses (no Ruby library is capable of parsing JSON as a stream).

Here is a sample of what node-logstash sends over:

{"message":"xxx","host":"hostA","@timestamp":"2013-12-19T00:09:03.995Z","@version":"1"}{"message":"yyy","path":"/tmp/logbench/1.log","host":"hostA","@timestamp":"2013-12-19T00:09:03.995Z","@version":"1"}{"message":"zzz", ... }

Strictly speaking, this is a logstash bug. However, it's not likely to be fixed soon, so in the meantime it would be nice to work around it by inserting a newline between traces, which doesn't introduce any problem anyway since it's JSON.

— Reply to this email directly or view it on GitHubhttps://github.com/bpaquet/node-logstash/issues/52 .

davux commented 10 years ago

I'd rather fix the json_logstash serializer – after all, it's called "logstash" because it's supposed to deliver logstash-friendly input. :)

Please see pull request #53 which should fix it. Sorry, I couldn't find how to attach a pull request to an issue.

bpaquet commented 10 years ago

I think you can not juste add a \n here. Some output does not need it.

I will look further for a more specific fix.

On Thu, Dec 19, 2013 at 10:02 PM, davux notifications@github.com wrote:

I'd rather fix the json_logstash serializer – after all, it's called "logstash" because it's supposed to deliver logstash-friendly input. :)

Please see pull request #53https://github.com/bpaquet/node-logstash/pull/53which should fix it. Sorry, I couldn't find how to attach a pull request to an issue.

— Reply to this email directly or view it on GitHubhttps://github.com/bpaquet/node-logstash/issues/52#issuecomment-30966190 .

davux commented 10 years ago

They don't need it, but newlines (whitespaces in general) are safe in JSON. In fact, JSON is often passed around with whitespaces here and there. It's actually its whole advantage for many people: the readability it provides.