search

evanphx / eventd-rfc

A RFC of a syslog replacement
54 stars 4 forks source link

Wording about logstash and message encodings #8

Open jordansissel opened 8 years ago

jordansissel commented 8 years ago

I was going to submit this as a PR to the readme, but I wasn't sure how to word the change, so I submit this for discussion.

From the readme:

  • Difficult (if possible at all) to dynamicly detect different message encodings
    • Meaning sending json or plain text over syslog isn't possible without treating the json as plain text

In this specific case (mixed json + plain text), logstash does support it. If you use codec => json (or json_lines) and invalid json is received, then it logs a the problem, treats the payload as plain text. and tags the event _jsonparsefailure. relevant from json codec here

Mixing json and plain text on a single input is fairly common, which is why we support this specific mix.

I admit that other mixed-encoding scenarios aren't supported, I'm not sure what data transport system would have detection of format for mixed content types without any hinting or handshakes to declare the serialization/encoding before transmission. For a hypothetical example of what isn't easy in Logstash. If you tried to mix protobuf, json, plain text, and msgpack on the same tcp input, there's no built-in way to achieve that encoding detection. This itself is a difficult problem with non-negotiated transports and is unrelated to Logstash.

Anyhooooooo. Short summary: Logstash handles mixed json+plain pretty well ;)

evanphx commented 8 years ago

Hi! Sure, that makes sense and we should clarify that in the README. What I was really speaking to is that embedding JSON in syslog means putting the JSON into a field that is freeform, and thus the decoder side has to be decoded with specific logic (like adding codec => json) in order to be understood.

I'll say also that this difficulty is really trivial in comparison to the other issues.