search

dwbutler / logstash-logger

Ruby logger that writes logstash events
MIT License
454 stars 118 forks source link

Character limit for json_lines #109

Closed clarkent86 closed 7 years ago

clarkent86 commented 7 years ago

I seem to be experiencing an issue with my rails app where I'm trying to log a large event and the json_lines codec seems to malfunction. When the event occurs successfully a lot of data is logged, but the log is sent to my catch all index instead of logging it under the name of the controller because instead of being parsed by json_lines it's all being dumped into the message field. When there is an issue with the application and not much is logged besides the failure, json_lines works fine.

dwbutler commented 7 years ago

Hi, thanks for the bug report.

I'll need to know more information to continue debugging this issue. Things like:

clarkent86 commented 7 years ago

I was pretty sure there was some sort of character limitation but as I've been working through my issue I've found it to be with the actual JSON format that was being parsed. I could only recreate it some of the time with the same test data; frankly it still doesn't make sense to me why it worked some of the time. Anyways I think I'm on the correct path to fixing my issue, sorry for the false alarm!

dwbutler commented 7 years ago

That would make sense. If you try to log a JSON string to LogStashLogger, and it cannot parse it, it will dump the contents into the message field (which is consistent with your observation).

clarkent86 commented 7 years ago

I've dug more into this issue. It's actually a logstash character limitation of 4095, so nothing to do with your code.

On Tue, Dec 20, 2016 at 6:16 PM, David Butler notifications@github.com wrote:

That would make sense. If you try to log a JSON string to LogStashLogger, and it cannot parse it, it will dump the contents into the message field (which is consistent with your observation).

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/dwbutler/logstash-logger/issues/109#issuecomment-268397796, or mute the thread https://github.com/notifications/unsubscribe-auth/AJOMx7dexi4yBzuZ35bAYK3TL2u2aIcVks5rKG_SgaJpZM4LSOtM .

-- -Joey Ross

dwbutler commented 7 years ago

What logstash listener configuration are you using? It would be good to know the details of your issue so it can be documented in the Troubleshooting section of the readme.

clarkent86 commented 7 years ago

I'm using logstash version 2.3.2. What other information would you want for documentation?

dwbutler commented 7 years ago

Basically the contents of the config file that you use with logstash. Or at least which output you're using so I can figure out where to look in the logstash source code.

clarkent86 commented 7 years ago

Here is my input code with the json_lines codec:

input { udp { host => "..." port => **** tags => ["rails"] codec => json_lines } }

It will show up dumped in the catch all index in the message field on kibana with a { after the 4095th character, without new lines. It is output to a redis queue, which might be the bottle neck I'm guessing? I'm not sure exactly, I'm basing this off of a consumable config. Anyways, :

output { if "rails" in [tags] { redis { host => 'localhost' data_type => 'list' key => 'rails:redis' congestion_threshold => 6000000 } } }

After that I have another logstash config set up to read in from that redis queue:

input { redis { host => "localhost" data_type => 'list' key => 'rails:redis' type => 'redis-input' } }

And that has a simple output to the Elasticsearch node that works fine for other events. It's indexed based on an if for the name of the controller and it has an else catch-all index.

dwbutler commented 7 years ago

I think you're experiencing the same issue reported in https://github.com/dwbutler/logstash-logger/issues/43. To the best of my knowledge, this is a bug in Logstash's implementation of the UDP input.

Is there a reason you're sending the events to a UDP listener first and then forwarding to Redis? Why not send the events directly from LogstashLogger to Redis?

dwbutler commented 7 years ago

You mentioned something that sounds important - the final { at the end of the message was not preceded by a newline. This is unexpected. This means that the newline character is being stripped at some point in the process.