search

logstash-plugins / logstash-output-lumberjack

Apache License 2.0
6 stars 24 forks source link

Lumberjack V2 output #27

Open robin13 opened 6 years ago

robin13 commented 6 years ago

This plugin currently communicates only using Lumberjack V1 output, but this is not compatible with the Lumberjack V2 protocol. This can be problematic if a custom application (not logstash-input-beats which speaks both V1 and V2) has been designed to work with the Lumberjack V2 protocol only. It would be great to have an option in the logstash-output-lumberjack plugin to choose the protocol version being used for communication.

webmat commented 6 years ago

Note to future implementer, Lumberjack protocol V1 is explained here.

According to @ph, V2 differences are:

Just to make sure we don't lose reference to this, the protocol defined here contains accurate information about v1. In short, V2 is replaced with V1 and the payload is in the form of key/value pair instead of using a JSON format. The ACK in V1 is a bit different: Do not support ACK0 keep alive. Do no support partial ACK Only ACK on full batch

webmat commented 6 years ago

@ph @jordansissel Is there a more formally defined spec for Lumberjack V2? Or is the Go implementation in Beats the only reference?

ntim commented 6 years ago

Lumberjack v2 also works better in combination with custom load balancers.

Furthermore, when configuring logstash-to-logstash communication as shown in the official docs, every minute or so, the sending logstash complains about a malformed frame.

ntim commented 6 years ago

@ph @jsvd @jordansissel any updates?

ntim commented 6 years ago

@webmat documentation of the protocol will not be done: https://github.com/elastic/libbeat/issues/279