search

dwbutler / logstash-logger

Ruby logger that writes logstash events
MIT License
456 stars 118 forks source link

TCP device loses events through timeout proxy #156

Open sheldonh opened 4 years ago

sheldonh commented 4 years ago

When the TCP device is used to send events to Logstash through a proxy that uses timeouts instead of TCP keepalives, it is guaranteed to lose events. A quiet client that logs only once per timeout period will observe 50% event loss.

I ran into this with https://logit.io/. They front Logstash with a proxy that uses application layer idle timeouts instead of TCP keepalives. I'm chatting to them about it, but meanwhile, here is a demonstration of the problem:

https://github.com/xneelo/elk-spike

I've had real trouble navigating the inheritance shenanigans to figure out what's going on, but it seems like the TCP device doesn't interact well with Connectable's retry strategy.

sheldonh commented 4 years ago

One filthy hack would be to emit blank lines periodically. The json_lines codec ignores blank lines.