search

JonZeolla / jzeolla-metron-bro-plugin-kafka

A Bro log writer plugin that sends logging output to Kafka.
Apache License 2.0
3 stars 4 forks source link

Default librdkafka message partitioner doesn't balance messages across partitions #2

Open dcode opened 6 years ago

dcode commented 6 years ago

When you accept the default partitioning scheme in librdkafka, it will randomly choose a partition at startup and fill only that one. Ideally, we can evenly distribute bro events across multiple partitions. This will allow for much better scaling.

It's possible that I'm overlooking something with the newer code in librdkafka. I've written some code to do this before, so I'll likely be sending you another PR if you don't fix it first.

JonZeolla commented 6 years ago

Good catch - would you mind opening a JIRA for this? There are definitely some performance improvements that can be made to the package, and we could use some more thoroughly reviewed documentation, both of which I hope to address in early 2018. That said, any contributions are very welcome!