Make it possible to use Cruise Control to move all data between two JBOD disks

[scholzj]

One of the common requests we get is to be able to easily move all Kafka logs between two disks on the JBOD storage array. This is useful for example when:

• Your current disk is too big and you want to use smaller one
• You watch to use different Storage Class with different parameters or different storage type

Currently, this needs ot be done using the Kafka CLI tools which is not really user-friendly. It would be nice to have an option to do it through Cruise Control and have CC handle moving the logs.

[ppatierno]

@scholzj right now we can do a rebalance across JBOD disks on the same broker. What exactly are you asking for? Not doing a rebalance but just moving ALL the logs from one disk to the other? I am not sure it's possible with Cruise Control, are you aiming for a proposal of having this feature on their community?

[kyguy]

Looking deeper at the CC wiki [1] it looks like we can move leader replicas off specific disks of a broker but can't specify where the data will go:

User can also request to move all the leader replicas away from the a list of disks via
POST /kafkacruisecontrol/demote_broker?brokerid_and_logdirs=[id1-logdir1, id2-logdir2...]

It doesn't fully meet the requested use case but I am sure we could this feature in upstream CC so that it could. That being said, we would also need to enable the demote_broker endpoint in Strimzi as well

[1] https://github.com/linkedin/cruise-control/wiki/REST-APIs#demote-a-list-of-brokers-from-the-kafka-cluster

[scholzj]

@ppatierno Basically, I'm looking for similar option to your mode: remove-brokers but for disks in the JBOD array. So it would move all data from given volume and make it empty so that it can be safely removed.

[scholzj]

Triaged on 12.1.2023: Does not seem to be supported in Cruise Control. We can open an issue in Cruise Control to track this as an enhancement there.

[scholzj]

Opened https://github.com/linkedin/cruise-control/issues/1958 in Cruise Control for this enhancement.

[teimyBr]

This will help us a lot.

Maybe we can push Linked In to complete the PR

https://github.com/linkedin/cruise-control/pull/1908

[scholzj]

Looks like the CC support has been merged now in https://github.com/linkedin/cruise-control/pull/1908. So maybe we can triage it again to see of something more is needed.

[scholzj]

Triaged on community call on 5.10.2023: A useful feature. A proposal should be done for the API changes etc.

[forsberg]

This would be very useful in the case where the usage pattern for your Kafka cluster on AWS has changed, and you find yourself with EBS volumes that are larger than what you need.

If I understand this feature correctly, it will allow you to attach new smaller JBOD EBS Volumes to an existing cluster, then move your data to the new volumes, and finally remove the old EBS volumes that can then be deleted, saving on cost.