search

cloudflare / goflow

The high-scalability sFlow/NetFlow/IPFIX collector used internally at Cloudflare.
BSD 3-Clause "New" or "Revised" License
859 stars 172 forks source link

protobuf messages in the length-delimited format #65

Closed bad3bs closed 4 years ago

bad3bs commented 4 years ago

It's will be awesome to add option to enable protobuf messages in the length-delimited format. That for native support clickhouse protobuf input (https://clickhouse.tech/docs/en/interfaces/formats/#protobuf).

lspgn commented 4 years ago

This is a good idea. Attached a PR. I'm not sure the protobuf can be mapped 1:1 within Clickhouse (example: storing src/dst IP addresses as bytes instead of two columns being a UInt32 and a FixedString(16)). But I tested locally (fed from a Kafka), it works.

lspgn commented 4 years ago

Added the functionality. I also updated flow-pipeline to provide a functioning example with Clickhouse.