This PR allows developers to configure Goka's partitioning behavior to match librdkafka. Open to feedback on the approach!
Why?
To allow the group table to be copartitioned with input topics whose producers use librdkafka's partitioning behavior. There is a subtle difference in the way librdkafka and Sarama convert hashes of keys to partition numbers (embarrassingly, here is my source for this claim). If a Goka processor processing an input topic with librdkafka-based producers uses sarama's partitioning behavior, the group table will not be copartitioned with the input topics.
Changes
Added a goka.ProducerBuilder constructor that allows developers more granular control over the producer's sarama.HashPartitioner, which is responsible for assigning keys to partitions.
Technically, you can already customize the sarama.HashPartitioner by writing a custom goka.ProducerBuilder, but I believe it would be nicer if Goka provided a first-party way to do this.
Added a view option goka.WithViewHashUnsigned that configures the view to calculate the number of the partition in which a key can be found in the same way librdkafka would.
This PR allows developers to configure Goka's partitioning behavior to match librdkafka. Open to feedback on the approach!
Why? To allow the group table to be copartitioned with input topics whose producers use librdkafka's partitioning behavior. There is a subtle difference in the way librdkafka and Sarama convert hashes of keys to partition numbers (embarrassingly, here is my source for this claim). If a Goka processor processing an input topic with librdkafka-based producers uses sarama's partitioning behavior, the group table will not be copartitioned with the input topics.
Changes
goka.ProducerBuilder
constructor that allows developers more granular control over the producer'ssarama.HashPartitioner
, which is responsible for assigning keys to partitions.sarama.HashPartitioner
by writing a customgoka.ProducerBuilder
, but I believe it would be nicer if Goka provided a first-party way to do this.goka.WithViewHashUnsigned
that configures the view to calculate the number of the partition in which a key can be found in the same way librdkafka would.