cloudflare / goflow

The high-scalability sFlow/NetFlow/IPFIX collector used internally at Cloudflare.
BSD 3-Clause "New" or "Revised" License
859 stars 172 forks source link

Length-delimited protobuf #65 #67

Closed lspgn closed 4 years ago

lspgn commented 4 years ago

Added -proto.fixedlen=true to have protobuf encoded with length before the message (Buffer.EncodeMessage). This is required by Clickhouse.

A table example is the following:

CREATE TABLE flows
(
    TimeReceived UInt64,
    SamplingRate UInt64,
    ...
) ENGINE = Kafka()
SETTINGS
    kafka_broker_list = 'kafka:9092',
    kafka_topic_list = 'flows',
    kafka_group_name = 'clickhouse',
    kafka_format = 'Protobuf',
    kafka_schema = './flow.proto:FlowMessage';