Adding action rebalance

[ssingudasu]

Introduction: To rebalance a topic in topicctl, We currently use topicctl apply $topic --rebalance / topicctl apply $topic --rebalance --to-remove . In a production scale Kafka cluster, there is a need to rebalance all topics for a cluster.

Goal:

• Add new action - rebalance to topicctl. i.e topicctl rebalance
• Print rebalance status / Metric information (stdout) during rebalance. Refer log [timestamp] INFO Metric: {}. Please refer below screenshots for output.
• Rebalance status / metrics information is two fold. overall topic rebalance status and topic rebalance status for each iteration

Considerations for action rebalance:

1. expects --cluster-config (i.e cluster.yaml) and --path-prefix (topics folder)
2. Uses underlying topicctl pkg/apply to rebalance a topic.

NOTE: A topic is not considered for rebalance if

• topic config is inconsistent with cluster config (name, region, environment etc...)
• partitions of a topic in kafka cluster does not match with topic partition setting in topic config
• retention.ms of a topic in kafka cluster does not match with topic retentionMinutes setting in topic config
• topic does not exist in kafka cluster

Test Ouput:

Tested on local kafka cluster (3 brokers, 1 in each rack) with 3 topics test-1 -> exists in kafka. But settings mismatch with test-1 topic yaml config test-2 -> exists in kafka. settings match (replication 2) test-3 -> does not exist in kafka

---

# go run cmd/topicctl/main.go -h

# go run cmd/topicctl/main.go rebalance -h

---

# go run cmd/topicctl/main.go rebalance --cluster-config $CLUSTER/cluster.yaml --path-prefix $CLUSTER/topics/ --auto-continue-rebalance --skip-confirm --partition-batch-size 5 --broker-throttle-mb 640

---

# go run cmd/topicctl/main.go rebalance --cluster-config $CLUSTER/cluster.yaml --path-prefix $CLUSTER/topics/ --auto-continue-rebalance --skip-confirm --partition-batch-size 5 --broker-throttle-mb 640 --to-remove 3

[petedannemann]

Please update the PR description to remove the examples where topics were being creating and add an example of what it looks when topics are rebalanced. A lot of logging was added and I wonder if things are too verbose now

[ssingudasu]

Please update the PR description to remove the examples where topics were being creating and add an example of what it looks when topics are rebalanced. A lot of logging was added and I wonder if things are too verbose now

PR description has been updated with detailed output screenshots along with test cluster and topic information

[petedannemann]

What do you think about doing checks for all the topics first, outputting a list of ones that will be skipped and why, prompting the user for (y/n) to continue based on that "plan", and then rebalancing the topics that have passed the checks? This way it's more obvious which topics actually failed to rebalance vs. which ones didn't attempt to rebalance at all

EDIT: I guess things can change between the plan and rebalance phase. Maybe this isn't a good idea

[ssingudasu]

Duplicate of https://github.com/segmentio/topicctl/pull/142

closing this