This adds support for removing brokers from replica assignment. We specify a replica removal by having a source-broker with no dest-broker. Replica addition could also be added in a similar way, but I'm gonna leave that for the next PR.
Before it was not possible because the script would do two steps of verification for replacement:
Check that the overall replication factor of the topic partitions has not changed. This check can now be skipped with the --rf-changes flag
Check that the replication factor of topic partitions within the same topic is the same. This check can now be skipped with the --rf-mismatch flag.
We also introduce a --topic-partition-filter flag, that is a path to some output from kafka-check, so that you can focus on fixing offline or specific partitions instead of generating a plan for all of them.
Wrote some unit tests and will be trying this out in dev later today, just want to get the review out first.
A sample invocation to generate a reassignment plan to remove a faulty broker id 0 might be:
This adds support for removing brokers from replica assignment. We specify a replica removal by having a source-broker with no dest-broker. Replica addition could also be added in a similar way, but I'm gonna leave that for the next PR.
Before it was not possible because the script would do two steps of verification for replacement:
--rf-changes
flag--rf-mismatch
flag.We also introduce a
--topic-partition-filter
flag, that is a path to some output fromkafka-check
, so that you can focus on fixing offline or specific partitions instead of generating a plan for all of them.Wrote some unit tests and will be trying this out in dev later today, just want to get the review out first.
A sample invocation to generate a reassignment plan to remove a faulty broker id
0
might be:kafka-cluster-manager -t test_cluster --write-to-file reassign_plan.json replace-broker --source-broker 0 --max-leader-changes 5000 --max-partition-movements 5000 --topic-partition-filter topic_partitions.txt --rf-mismatch --rf-change
where
topic_partitions.txt
is similar tokafka-check
output:While the above command is rather long, it should be much better than relying on adhoc scripts to generate reassignment json files.