search

Blizzard / node-rdkafka

Node.js bindings for librdkafka
MIT License
2.11k stars 392 forks source link

Cooperative Rebalance #1081

Open serj026 opened 3 months ago

serj026 commented 3 months ago

The purpose of these changes is to add a functional version of node-rdkafka that supports incremental cooperative rebalancing. Given that librdkafka has supported this type of rebalancing for nearly four years, this update aims to bring node-rdkafka in line with librdkafka capabilities and enhancing its functionality.

We use the code from this fork in our production environment for services that handle 5000 requests/messages per second. Using the cooperative-sticky partitioning strategy has shown good results by avoiding "stop-the-world" situations. Specifically, we have observed increased throughput and reduced spikes during scale-up and scale-down operations of pods in Kubernetes.

SeanReece commented 1 month ago

@GaryWilber Any way we could get some eyes on cooperative rebalancing? :) This would be huge for us.

GaryWilber commented 1 month ago

@iradul Do you have access to trigger the PR test checks/workflow for this? I don't see a way to trigger it myself.

ZhukovAntonPls commented 1 month ago

@SeanReece @GaryWilber @iradul Guys, how can we escalate the testing and merging process of this PR? It would be very useful for the whole community