search

linkedin / cruise-control

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.
https://github.com/linkedin/cruise-control/tags
BSD 2-Clause "Simplified" License
2.74k stars 587 forks source link

Execution with throttle set got failed if topic is removed during execution #1350

Closed jiao-zhangS closed 3 years ago

jiao-zhangS commented 3 years ago

Hi, team

We hit the problem when calling remove_broker API with 'replication_throttle' set. The reassignment execution failed at "clearThrottles". The exception is as following.

[2020-10-08 18:11:46,468] ERROR Executor got exception during execution (com.linkedin.kafka.cruisecontrol.executor.Executor)
kafka.admin.AdminOperationException: Topic "test-v1" does not exist.
    at kafka.zk.AdminZkClient.validateTopicConfig(AdminZkClient.scala:326)
    at kafka.zk.AdminZkClient.changeTopicConfig(AdminZkClient.scala:340)
    at com.linkedin.kafka.cruisecontrol.executor.ExecutorUtils$.changeTopicConfig(ExecutorUtils.scala:135)
    at com.linkedin.kafka.cruisecontrol.executor.ExecutorUtils.changeTopicConfig(ExecutorUtils.scala)
    at com.linkedin.kafka.cruisecontrol.executor.ReplicationThrottleHelper.removeThrottledReplicasFromTopic(ReplicationThrottleHelper.java:224)
    at java.util.HashMap.forEach(HashMap.java:1289)
    at com.linkedin.kafka.cruisecontrol.executor.ReplicationThrottleHelper.clearThrottles(ReplicationThrottleHelper.java:113)
    at com.linkedin.kafka.cruisecontrol.executor.Executor$ProposalExecutionRunnable.interBrokerMoveReplicas(Executor.java:1183)
    at com.linkedin.kafka.cruisecontrol.executor.Executor$ProposalExecutionRunnable.execute(Executor.java:994)
    at com.linkedin.kafka.cruisecontrol.executor.Executor$ProposalExecutionRunnable.run(Executor.java:920)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

It's a bug, right? btw the version in use was 2.4.23.

efeg commented 3 years ago

@jiao-zhangS Thanks for reporting. Correct -- this is a duplicate of the issue https://github.com/linkedin/cruise-control/issues/1318.