search

bsycorp / log-forwarder

A simple log forwarder for systemd, docker and kubernetes logs
Apache License 2.0
6 stars 0 forks source link

sumologic fails to properly detect message boundaries at certain log volume #9

Open nhoughto opened 5 years ago

nhoughto commented 5 years ago

It appears sumologic, at a certain log volume to a single HTTP collector will start misbehaving and instead of detecting bounds of a log message via timestamp detection (or however it works) it creates a sumologic message for every line (ish?) submitted to the HTTP collector.

There don't appear to be any knobs to turn in sumologic config to change it, and there appears to be a threshold we've crossed to see this behaviour, it previously worked ok (and appears to work if log volumes drop enough).

Maybe we should be balancing emitted logs across multiple HTTP collector URLs? 😬

nhoughto commented 5 years ago

@bls thoughts on supporting multiple urls for each type and balancing across them?

bls commented 4 years ago

🤮

nhoughto commented 4 years ago

something to discuss with your friendly neighbourhood sumo repo maybe, rather than guessing / avoiding it. Pretty shitty it works that way tbh.

Otherwise once fluentbit is the upstream way were supposed to do it, but it uses the same HTTP collector, AND doens't support proper tagging/headers as per: https://github.com/fluent/fluent-bit/issues/1213

So kinda hosed either way, need a collector that reads journals that uses the proper java collector api.