search

trivago / gollum

An n:m message multiplexer written in Go
http://gollum.readthedocs.org/en/latest/
Apache License 2.0
940 stars 74 forks source link

Feature request: Allow overriding Syslog RFC protocols #234

Closed jabinb closed 6 years ago

jabinb commented 6 years ago

When using the Syslog consumer with < RFC6587 you get forced into using UDP for the transport due to strict adherence to the standard. However while correct this prevents you from accepting messages from non-rfc compliant loggers that use TCP.

This is born from a situation I'm currently dealing with using IBM Kubernetes Service where they ship remote logs via dlackty/fluent-plugin-remote_syslog. However payload sizes >1kb get truncated for UDP packets so I'm forced to use TCP. This would be fine but dlackty/fluent-plugin-remote_syslog actually creates syslog messages using RFC3164 which is defined as UDP only so gollum forces the transport from TCP -> UDP and I lose large messages (these are http request logs that can have lengthy json).

The options I've tried to work around this are:

  1. Make a proxy-consumer on tcp://0.0.0.0:514 and a producer to unixgram:///var/gollum/proxy then create the RFC3614 listener on that unix socket, however I seem to get a lot of I/O read errors and drop messages but this might just be my bad config.
  2. Make a TCP socket and attempt to grok the format myself, I dropped this idea as it was more work than 3.
  3. Patch gollum to remove the forcing to udp which works well, I receive the messages as expected but it feels dirty to have to hack the code and produce my own docker image.

While I understand the why the forcing to UDP is done but I think for Gollum's goal of being a multiplexer with many different flexible input/outputs the strict adherence to the RFCs without the possibility of overriding them limits ingesting messages from sources that might not be RFC compliant and forces you down the path of doing weird spaghetti workarounds.

Here's my proposals on solutions:

  1. Remove the TCP -> UDP coercion and make the warning loud and obvious.
  2. Introduce an explicit parameter that allows you to force the protocol ignoreRFC: true.
  3. If it is possible come up with an accepted example config for re-routing/proxying TCP syslog traffic to a UDP consumer
arnecls commented 6 years ago

Hey there. First: The proxy consumer / producer is more like a proof of concept, so nothing I would personally use in production.

As for the feature request - should be doable, I'll have a look at it. Do you requires this for 0.5.x or is it ok to integrate it into 0.6 (current master) only? Just to be clear - there is no planned release date for 0.6, yet so you would probably have to "live off master" for a quite while.

jabinb commented 6 years ago

Hey @arnecls,

Yep I kinda got that feeling from using the proxy, it felt I was abusing it too trying to get it to do what I wanted.

It's okay to push this into 0.6, I'll pin my images to master when it's merged and update once 0.6 rolls out. Really appreciate the quick response, thanks for all your help!

arnecls commented 6 years ago

fix has been merged into master.