search

cloudflare / goflow

The high-scalability sFlow/NetFlow/IPFIX collector used internally at Cloudflare.
BSD 3-Clause "New" or "Revised" License
879 stars 175 forks source link

Add support for specifying Kafka key #15

Closed pksu closed 5 years ago

pksu commented 5 years ago

Introduce a new command line switch to enable string based Kafka key (that is disabled by default).

lspgn commented 5 years ago

We can add this, I am just wondering your use case: do you need to send flows to a specific partition? I would imagine keying via specific attributes (eg: router address).

simPod commented 5 years ago

Personally, I'd use this to partition messages by device (so eg. use sflow listening port).

simPod commented 5 years ago

Just stumbled upon the use case where I need to identify port through inif/outif + device. I could extract device identifier from key if it was there. Currently, there's no other way to identifiy device when consuming from kafka I think.

EDIT: "router addr" field

lspgn commented 5 years ago

I think I will add the flag to set a key using the proto fields (which could also take something static). Example: -kafka.key example-%inif-%outif

simPod commented 5 years ago

That would IMO solve every issue I can come up with right now 👍

lspgn commented 5 years ago

Overdue but added the feature in GoFlow v3: https://github.com/cloudflare/goflow/blob/master/transport/kafka.go#L49

Sorry for taking so long to reply. Thanks for your contribution. It helped me.