Open aratz-lasa opened 7 months ago
twmb
commented
6 months ago I have to evaluate what this entails. The branch is stale because I used to implement protocol support before it was rolled out in Kafka, but stopped once Kafka started changing APIs repeatedly before the API was finally released.
I'm not sure 848 is high priority though because it's released as preview only. I'm not sure how hard it'll be to implement, but likely pretty hard...
aratz-lasa
commented
6 months ago Yeah, it's pretty hard. I've been reading all the JIRA tasks they have, and it's crazy. They made significant changes to the client. Here you can see notes about the new client-side state machine: https://cwiki.apache.org/confluence/display/KAFKA/Consumer+rebalance#Consumerrebalance-RebalanceStateMachine
Most of the remaining tasks will be merged for 3.8.0 release (which will be in May). However, I didn't find any official source on the timeline.
twmb
commented
6 months ago Yeah I'm probably going to need help with this, this is a big one.
Otherwise this may be unsupported for a while while I slowly slowly tinker away adding support. It seems like a lot of work for minor gain.
aratz-lasa
commented
5 months ago @twmb I'm able to help you. This week, I'm starting to implement the server-side logic, so I'll get a good understanding of the topic. If you want we can collaborate on this work.
twmb
commented
3 months ago I definitely missed your reply above^.
I'm starting to think about this (a little bit). One thing I may lean to do first is to improve the guts of the consumer so that moving a partition between brokers does not require a full consumer stop/restart -- and then after, implement KIP-848.
Or it may make sense to just start with 848 logic, I'm really not sure -- I haven't dug into it much.
I wanted to know what is the plan on implementing KIP-848. I see a dedicated branch just for this, but it's been inactive for a year. @twmb I would be happy to help move it forward.