search

twmb / franz-go

franz-go contains a feature complete, pure Go library for interacting with Kafka from 0.8.0 through 3.7+. Producing, consuming, transacting, administrating, etc.
BSD 3-Clause "New" or "Revised" License
1.78k stars 182 forks source link

Direct consumer doesn't implement KIP-392 #817

Closed soundvibe closed 3 weeks ago

soundvibe commented 3 weeks ago

Fetching from closest replicas doesn't work when using direct consumer (via kgo.ConsumePartitions()). This works when switching to group consumer but using it isn't convenient and practical for some use cases when static membership is needed.

twmb commented 3 weeks ago

The area of the code that handles KIP-392 is no different between the group consumer and the direct consumer. The group consumer, in a way, is a layer on top of the direct consumer (managing the directness).

KIP-392 is opted into via the Rack option, and that's it.

Do you have logs of this failing to not happen on the direct consumer, but happening on the group consumer? The most recent release contains a log line that shows when preferred replica application is happening: https://github.com/twmb/franz-go/commit/cda897d

soundvibe commented 3 weeks ago

Yes, I'm getting fetch partitions returned preferred replicas in the logs when using group consumer but this isn't logged when using direct consumer.

soundvibe commented 3 weeks ago

After digging deeper it seems that rack awareness works with direct consumer when offsets are set using NewOffset(). Previously, I was setting offsets which were fetched using admin package, so I guess they had some internal state about which broker to use.