search

twmb / franz-go

franz-go contains a feature complete, pure Go library for interacting with Kafka from 0.8.0 through 3.7+. Producing, consuming, transacting, administrating, etc.
BSD 3-Clause "New" or "Revised" License
1.78k stars 182 forks source link

Inconsistent partitioning strategy #680

Closed akshatraika-moment closed 7 months ago

akshatraika-moment commented 7 months ago

Does kgo have special partitioning logic if we are opting for default key based partitioning? I have 2 services that both write to the same kafka topic. One uses kgo and one uses confluent-kafka-go and the messages are ending up in different partitions.

twmb commented 7 months ago

It shouldn't, the default partitioner uses the same logic, including murmur2 hashing. I don't know how to run (i.e. test) Java but I did transliterate the Java code directly. You're using records with the same key?

yianni commented 7 months ago

@akshatraika-moment, I think the Confluent client uses CRC32 by default as it's based on librdkafka. Take a look at https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md#topic-configuration-properties and search for 'partitioner'.

twmb commented 7 months ago

Ah, I would've figured confluent-kafka-go mirrored the Java codebase. ty @yianni.

If that's the case -- SaramaHasher with crc32 I think mirrors librdkafka, per #669

akshatraika-moment commented 7 months ago

I see, I'll try that. Appreciate the quick response!

twmb commented 7 months ago

Let me know this isn't resolved, closing now

akshatraika-moment commented 7 months ago

I was able to resolve this by switching the hash fn. thanks!