lovoo / goka

Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go.
BSD 3-Clause "New" or "Revised" License
2.36k stars 173 forks source link

Add options for librdkafka partitioning compatibility #455

Open owenniles opened 3 months ago

owenniles commented 3 months ago

This PR allows developers to configure Goka's partitioning behavior to match librdkafka. Open to feedback on the approach!

Why? To allow the group table to be copartitioned with input topics whose producers use librdkafka's partitioning behavior. There is a subtle difference in the way librdkafka and Sarama convert hashes of keys to partition numbers (embarrassingly, here is my source for this claim). If a Goka processor processing an input topic with librdkafka-based producers uses sarama's partitioning behavior, the group table will not be copartitioned with the input topics.

Changes

owenniles commented 3 months ago

It occurred to me that setting the partitioning behavior for each lookup table may be desirable as well https://github.com/lovoo/goka/issues/456.