cloudflare / goflow

The high-scalability sFlow/NetFlow/IPFIX collector used internally at Cloudflare.
BSD 3-Clause "New" or "Revised" License
859 stars 172 forks source link

add xtra flag to send json on Kafka transport #78

Open gjelu opened 4 years ago

gjelu commented 4 years ago

The protobuffer has support for marshaling protobuffer messages as json: google.golang.org/protobuf/encoding/protojson

For some popular tools, like logstash (that are more text oriented), having json would be a better option as message format. (See https://github.com/logstash-plugins/logstash-codec-protobuf/issues/15). At least it will make it a lot easier to share data between (legacy) systems.

Basically the impact would be minimum: in the Kafka transport the protobuffer message is marshaled as json format instead of wire format based on a flag. Fields of type bytes are then converted as base64 strings which is, for some systems, safer.

I've implemented a working version for this: https://github.com/gjelu/goflow/commit/e2a0c56de200a6dc99f69ab3561efd4c280b3b51

Some extra required changes:

I can do a PR if it's feasible that this can be accepted? Or if you have any extra comments on this?

Slepwin commented 3 years ago

It would be very useful feature for me.

mugugnu commented 3 years ago

I'd love to see this feature!