search

netsampler / goflow2

High performance sFlow/IPFIX/NetFlow Collector
BSD 3-Clause "New" or "Revised" License
487 stars 112 forks source link

bug: fix kafka transport default partitioning #246

Closed lspgn closed 11 months ago

lspgn commented 11 months ago
simPod commented 11 months ago

I have tried to deploy the binary built from this PR but it seems to have no effect 🤔 Messages are still produced to only a single partition. I don't use -mapping so I suppose the key is nil.

image
simPod commented 11 months ago

I can see it is being produced with a key

image
simPod commented 11 months ago

I have tried a build where I removed this line https://github.com/netsampler/goflow2/blob/3c00d97a67c8c84886fe4b60fc457381cc46183d/transport/kafka/kafka.go#L233 and now it's distributed randomly

image

Unfortunately, I have no way to debug where is the key being created at the moment. I've requested tcpdump installation on the server where the sflow is being sent to so I can capture some packets, create sflow fixtures for goflow and debug it.

lspgn commented 11 months ago

Could you check again @simPod?

~I think I found the problem: empty key had a fixed hash which didn't work with the RoundRobinPartitioner.~

I'm observing it working with the expected behavior even when key is fixed. Make sure -transport.kafka.hashing=false

For reference in Bitnami Kafka docker:

Create a topic with 9 partitions

$ /opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic flow-messages --partitions 9

Get the offsets

$ /opt/bitnami/kafka/bin/kafka-get-offsets.sh --topic flow-messages --bootstrap-server localhost:9092
flow-messages:0:1506
flow-messages:1:1506
flow-messages:2:4009
flow-messages:3:1506
flow-messages:4:1506
flow-messages:5:1506
flow-messages:6:8212
flow-messages:7:1506
flow-messages:8:1504
simPod commented 11 months ago

I can confirm it works without enabling -transport.kafka.hashing. This can be now closed I guess https://github.com/netsampler/goflow2/issues/248