search

IBMStreams / streamsx.kafka

Repository for integration with Apache Kafka
https://ibmstreams.github.io/streamsx.kafka/
Apache License 2.0
13 stars 9 forks source link

upgrade kafka-clients from 2.3.1 to 2.5.1 #217

Closed ghost closed 3 years ago

ghost commented 4 years ago

The clients component of kafka-2.5.1 has some significant fixes.

The toolkit must implement the new onPartitionsLost(partitions) function of the ConsumerRebalanceListener interface. For use in consistent region, a reset of the region would be the appropriate handling. When not in consistent region, at least the offset manager and the queue must be cleaned up from the offsets and records of the lost partitions.

ghost commented 4 years ago

Special care must also be taken when the incremental cooperative rebalancing protocol is chosen together with a sticky assignor: The semantics of the onPartitionsAssigned(partitions) callback changes: with incremental cooperative rebalancing, this callback hands only the new set of partitions since last rebalance, whereas the eager rebalance protocol hands the complete set of assigned partitions.

For use in consistent region, always the eager rebalancing protocol (stop-the-world) must be configured via the partition assignment strategy as incremental revocations and assignments of partitions would trigger multiple consistent region resets during rebalancing.

For use in autonomous region, the consumers should be able to take advantage from the incremental rebalancing introduced with Kafka 2.4, KIP-429.

ghost commented 4 years ago

This issue deals with upgrading kafka-clients only.

For the issues arising from new rebalancing functions available through the new client, #218 has been opened.