search

awslabs / flowgger

A fast data collector in Rust
BSD 2-Clause "Simplified" License
834 stars 55 forks source link

TCP/Tls output #2

Closed pdepaepe closed 9 years ago

pdepaepe commented 9 years ago

Awesome code, thanks a lot.

May you please consider to offer also a TCP/TLS output?

thomas-maurice commented 9 years ago

+1, that feature would actually be awesome

jedisct1 commented 9 years ago

Is the intent to forward Flowgger's output to another Flowgger instance?

Sticking to existing formats currently means having to use GELF since we have to retain data types. But JSON parsing/generation is quite CPU intensive and would be a pretty poor choice for inter-Flowgger communication.

Internally, records are stored as a structure that we can send to a downstream server with minor overhead using Flatbuffers or Cap'n Proto. Using these also means that modern applications can leverage them as well, and parsing them is bound to be insanely faster than LTSV.

This requires a few changes. Namely, we have to support binary formats, which don't fit really well with distinct splitting/decoding steps. We need either the splitter or the decoder to perform both steps. And of course, if the intent is to send data to another Flowgger server, we need to implement Flatbuffers or Cap'n Proto both for input and for output.

The good news is that all of these have been done. Cap'n Proto is now fully supported.

pdepaepe commented 9 years ago

No, Flowgger to Graylog TCP/TLS Input Gelf.

jedisct1 commented 9 years ago

Directly sending GELF over to Graylog brings requires extra changes, since individual messages have to be changed differently according to the backend. Graylog wants \0 after each message, \n makes more sense for the debugging output, Kafka uses one log message per Kafka message and shouldn't have the \0.

So, a new step has been added to the processing chain, in order to let the user pick how individual messages should be wrapped after having been encoded.

See https://github.com/jedisct1/flowgger/wiki/Mergers

jedisct1 commented 9 years ago

I still need to update the Wiki, but the TLS output should now be usable.