search

linkedin / cruise-control

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.
https://github.com/linkedin/cruise-control/tags
BSD 2-Clause "Simplified" License
2.74k stars 587 forks source link

Support for Connecting Kafka Cruise Control to Multiple Kafka Clusters #2177

Closed Shreyank031 closed 1 month ago

Shreyank031 commented 2 months ago

Hello,

I am using Kafka Cruise Control on a single EC2 instance and would like to connect it to multiple Kafka clusters hosted on Amazon MSK. While I am able to successfully connect to a single Kafka cluster, I haven't found documentation or guidance on how to set up Cruise Control to manage multiple clusters simultaneously.

Specifically, I need to manage three Kafka clusters: staging, prod-replica, and prod. Each cluster has its own bootstrap.servers and zookeeper.connect settings.

I have tried setting up separate cruisecontrol.properties files and running Cruise Control on different Web UI ports (e.g., 9091, 9092) for each cluster, but I'm not sure if this is the best approach.

Here are some additional details:

Is there a recommended way to configure Cruise Control to manage multiple Kafka clusters in this scenario? Any guidance or documentation references would be greatly appreciated.

Thank you!

HonestTelevision commented 1 month ago

I'm not sure if it's possible for a single cruise control instance to manage multiple kafka clusters, one alternative is to run one cruise control instance for each kafka cluster you have, and then use cruise-control-ui as a unified interface for all the cruise control instances.

Shreyank031 commented 1 month ago

@HonestTelevision Firstly thanks for replying. You are right. Three curise-control instances connected to three kafka clusters with unified curise-control-ui is the way. I am not sure if it's recommended to run all three curise-control in a single instance with docker. And then unify it with cruise-control-ui. It is a hassle, I am not sure whether it's going to work.

HonestTelevision commented 1 month ago

I think running three cruise control instances on one docker container it might work (though maybe things like env variables will be messed up), but it's probably not recommended. Why not just run three separate containers each with one instance?

Shreyank031 commented 1 month ago

Yes. I meant 3 containers in a single instance. And prometheus running in the instance.

mhratson commented 1 month ago

CruiseControl is not designed to manage multiple cluster at once. Please use separate CC instance for each cluster.