search

linkedin / cruise-control

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.
https://github.com/linkedin/cruise-control/tags
BSD 2-Clause "Simplified" License
2.76k stars 592 forks source link

Does CC have plans to take advantage of KIP-660 (Pluggable ReplicaAssignor)? #1338

Open tomncooper opened 4 years ago

tomncooper commented 4 years ago

KIP-660: Pluggable ReplicaAssignor has been proposed. This will allow a custom assigner to be used on topic creation rather than waiting for Kafka to make the decision and then having to rebalance.

If Cruise Control could be used as the assigner then it would allow the creation of topic partitions on under=utilized nodes rather than heavily loaded ones and significantly reduce the number of rebalances needed over the lifetime of a Kafka Cluster.

This is a pretty recent KIP so there is a chance for CC to shape the design of this (to some degree).

efeg commented 4 years ago

@tomncooper Thanks for sharing the KIP -- I agree that CC can help with the initial assignment of partitions. I commented on the relevant thread in kafka-dev mailing list.