Add protocol spec/description

[urso]

Reference: urso/go-lumber-old#4

There is a protocol description for the old Lumberjack protocol at https://github.com/logstash-plugins/logstash-input-beats/blob/v2.0.0/PROTOCOL.md but I'm missing like this for the Beats protocol.

While having a working library for the Lumberjack/Beats protocol is nice and a specification could be "reverse engineered" by looking at the Go code, it would be great if a textual protocol spec of the Lumberjack (v1) and Beats (v2) protocols existed.

[trixpan]

+1

This has been requested a while ago:

https://github.com/elastic/libbeat/issues/279

[joschi]

@urso Any news here?

[xfournet]

It is needed to ensure that other implementation are correct, eg https://github.com/apache/camel/tree/master/components/camel-lumberjack

[trixpan]

@jordansissel now that 5.0 is out, any chances of getting some traction over this? Amazing opportunity to enlarge the ELK ecosystem and yet this seems to be super low priority?

[jordansissel]

@trixpan I don't think the beats protocol is "done" yet, so I personally, I don't really want to document what isn't necessarily finalized. I'd prefer not to end up with a situation where the protocol/etc is documented, but we keep up-revisioning things and breaking things for everyone else -- Kafka and AMQP both do this, for example, with incompatible network changes on upgrades, and it's extremely frustrating.

If you really want to write your own client, today, you can start with the old doc which I wrote and referenced frequently when I needed to verify things - https://github.com/elastic/logstash-forwarder/blob/master/PROTOCOL.md.

[joschi]

@jordansissel I'd argue that the Lumberjack and Beats 1.x protocols are in fact "done".

There are lots of applications out there (*beats 1.x, logstash-forwarder, lumberjack, Logstash) speaking exactly these protocols and not having the wire protocols properly specified hurts 3rd party implementers. 😞

If you really want to write your own client, today, you can start with the old doc which I wrote and referenced frequently when I needed to verify things - https://github.com/elastic/logstash-forwarder/blob/master/PROTOCOL.md.

Unfortunately that spec is missing JSON frames and is using the "wrong" version number, compared with Beats protocol.

[trixpan]

@jordansissel I understand your concerns but also agree with @joschi : The protocol is already defined. Fluid or not, it is implemented by reverse engineering across many projects, but while data framing is largely sorted, expected behavior is still not generally agreed, leading to this sort of posts:

https://github.com/elastic/libbeat/issues/279#issuecomment-193373322

[andrewthad]

Three years later, is the lumberjack protocol "done" yet? Or is it still in flux?