search

linkedin / cruise-control

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.
https://github.com/linkedin/cruise-control/tags
BSD 2-Clause "Simplified" License
2.74k stars 587 forks source link

Replication throttler removes quotas for brokers having in-progress tasks/reassignments #1511

Closed amuraru closed 3 years ago

amuraru commented 3 years ago

We are seeing an unexpected behaviour when setting rebalance throttle. Here is a sequence of events where CC generated a reassignment plan for 4 replicas (2 for topicA and 2 for topicB).

2021-04-06T02:47:54.422431852Z 51908640 [posalExecutor-0] INFO utor.ReplicationThrottleHelper - Setting a rebalance throttle of 209715200 bytes/sec
2021-04-06T02:47:54.421718697Z 51908639 [posalExecutor-0] INFO ruisecontrol.executor.Executor - Starting executing balancing proposals.
2021-04-06T02:47:54.422274371Z 51908640 [posalExecutor-0] INFO ruisecontrol.executor.Executor - Starting 4 inter-broker partition movements.
2021-04-06T02:47:54.422427451Z 51908640 [posalExecutor-0] INFO ruisecontrol.executor.Executor - Executor will execute 4 task(s)
021-04-06T03:06:57.162657774Z 53051380 [posalExecutor-0] INFO ruisecontrol.executor.Executor - Finished tasks: [{EXE_ID: 25, INTER_BROKER_REPLICA_ACTION, {topicA-7, oldLeader: 307, [307, 106, 204, 207] -> [307, 106, 204, 101]}, COMPLETED}].
2021-04-06T03:06:57.162674863Z 53051380 [posalExecutor-0] INFO ruisecontrol.executor.Executor - 1/4 (25.00%) inter-broker partition movements completed. 31261/580279 (5.39%) MB have been moved.
2021-04-06T03:06:57.162678359Z 53051380 [posalExecutor-0] INFO utor.ReplicationThrottleHelper - Removing replica movement throttles from brokers in the cluster: []
2021-04-06T03:06:57.169505343Z 53051387 [posalExecutor-0] INFO ruisecontrol.executor.Executor - Executor will execute 0 task(s)
2021-04-06T03:07:07.190599636Z 53061408 [posalExecutor-0] INFO ruisecontrol.executor.Executor - Finished tasks: [{EXE_ID: 28, INTER_BROKER_REPLICA_ACTION, {topicA-27, oldLeader: 307, [307, 204, 206, 106] -> [307, 204, 101, 106]}, COMPLETED}].
2021-04-06T03:07:07.1906179Z 53061408 [posalExecutor-0] INFO ruisecontrol.executor.Executor - 2/4 (50.00%) inter-broker partition movements completed. 62435/580279 (10.76%) MB have been moved.
2021-04-06T03:07:07.190621743Z 53061408 [posalExecutor-0] INFO utor.ReplicationThrottleHelper - Removing replica movement throttles from brokers in the cluster: [101, 106, 204, 206, 307]
2021-04-06T03:07:07.221649883Z 53061439 [posalExecutor-0] INFO ruisecontrol.executor.Executor - Executor will execute 0 task(s)

^^^ ReplicationThrottleHelper is removing the movement throttles from broker e.g 101 although this broker is still involved in another reassignment task (26) - see below.


2021-04-06T03:27:09.940118098Z 54264158 [posalExecutor-0] INFO ruisecontrol.executor.Executor - Finished tasks: [{EXE_ID: 26, INTER_BROKER_REPLICA_ACTION, {topicB-57, oldLeader: 207, [207, 107, 308] -> [207, 101, 308]}, COMPLETED}].
2021-04-06T03:27:09.940138712Z 54264158 [posalExecutor-0] INFO ruisecontrol.executor.Executor - 3/4 (75.00%) inter-broker partition movements completed. 321331/580279 (55.38%) MB have been moved.
2021-04-06T03:27:09.940162612Z 54264158 [posalExecutor-0] INFO utor.ReplicationThrottleHelper - Removing replica movement throttles from brokers in the cluster: [101, 107, 207, 308]
2021-04-06T03:27:09.96490809Z 54264182 [posalExecutor-0] INFO ruisecontrol.executor.Executor - Executor will execute 0 task(s)
2021-04-06T03:33:30.821990596Z 54645039 [posalExecutor-0] INFO ruisecontrol.executor.Executor - Finished tasks: [{EXE_ID: 27, INTER_BROKER_REPLICA_ACTION, {topicBN-130, oldLeader: 208, [208, 104, 301] -> [208, 101, 301]}, COMPLETED}].
2021-04-06T03:33:30.822010692Z 54645039 [posalExecutor-0] INFO ruisecontrol.executor.Executor - 4/4 (100.00%) inter-broker partition movements completed. 580279/580279 (100.00%) MB have been moved.
2021-04-06T03:33:30.822014771Z 54645039 [posalExecutor-0] INFO utor.ReplicationThrottleHelper - Removing replica movement throttles from brokers in the cluster: [101, 104, 208, 301]
2021-04-06T03:33:30.841760371Z 54645059 [posalExecutor-0] INFO ruisecontrol.executor.Executor - Inter-broker partition movements finished.
2021-04-06T03:33:30.841795675Z 54645059 [posalExecutor-0] INFO ruisecontrol.executor.Executor - Starting 0 intra-broker partition movements.
2021-04-06T03:33:30.84180364Z 54645059 [posalExecutor-0] INFO ruisecontrol.executor.Executor - Intra-broker partition movements finished.
2021-04-06T03:33:30.841818071Z 54645059 [posalExecutor-0] INFO ruisecontrol.executor.Executor - Starting 0 leadership movements.
2021-04-06T03:33:30.841822497Z 54645059 [posalExecutor-0] INFO ruisecontrol.executor.Executor - Leadership movements finished.
2021-04-06T03:33:30.841858001Z 54645059 [posalExecutor-0] INFO operationLogger - Task [7d2fc772-4061-4fb3-a5ce-f793c79657b1] self-healing execution is finished.
2021-04-06T03:33:30.841886591Z 54645059 [posalExecutor-0] INFO ruisecontrol.executor.Executor - Execution finished.

Attaching also the replication throttle rates as reported by Kafka during these reassignments triggered by CruiseControl: (kafka_server_followerreplication_byte_rate and kafka_server_leaderreplication_byte_rate metrics) image

Here we can see that broker-101 follower incoming byte rate was higher than the configured quota (200MB/s)

efeg commented 3 years ago

@amuraru Thanks for reporting the issue with details (and sorry for the delayed response here)! The removal of throttle from broker 101 indeed seems premature.

Would you be interested in contributing a fix for this issue?