self healing for one broker failure ran two times

[jiao-zhangS]

Hi, we are using 2.4.36 and enabled 'self.healing.broker.failure.enabled'. We observed when one broker failed, self healing was executed after 'broker.failure.self.healing.threshold.ms'(we set it as 5 mins). And self healing was triggered again(out of expectation) after first time execution is finished. We are suspecting the same anomaly was put back to queue again as here's code shows because 'skipReportingIfNotUpdated' set as 'false' here. Is my suspect correct? any clues why two times execution occurred for one broker's failure. Another suspicious point is why the same log are logged twice almost at the same timing..

Following are some logs which can help describe what happened.

• 2021-04-15 18:13:15,893: Self-healing started successfully
• 2021-04-15 18:51:33,116: Execution finished

  [2021-04-15 18:08:13,970] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} for CHECK_WITH_DELAY. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:08:13,970] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} for CHECK_WITH_DELAY. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:10:13,970] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} for CHECK_WITH_DELAY. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:10:13,970] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} for CHECK_WITH_DELAY. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:13:15,893] INFO [e7874227-8ceb-40a2-847e-e9f7d84fef44] Self-healing started successfully. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:13:15,970] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} because executor is in INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS state. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:13:15,970] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} because executor is in INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS state. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:18:16,037] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} because executor is in INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS state. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:18:16,037] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} because executor is in INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS state. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:23:16,126] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} because executor is in INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS state. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:23:16,126] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} because executor is in INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS state. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:28:16,196] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} because executor is in INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS state. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:28:16,196] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} because executor is in INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS state. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:33:16,280] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} because executor is in INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS state. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:33:16,280] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} because executor is in INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS state. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:38:16,352] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} because executor is in INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS state. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:38:16,352] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} because executor is in INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS state. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:43:16,420] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} because executor is in INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS state. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:43:16,420] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} because executor is in INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS state. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:48:16,482] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} because executor is in INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS state. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:48:16,482] INFO Post processing anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}} because executor is in INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS state. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:51:33,116] INFO Execution finished. (com.linkedin.kafka.cruisecontrol.executor.Executor)

  After the execution was finished, we saw a lot of following 'Generating a fix for the anomaly' log.

  [2021-04-15 18:53:16,600] INFO Generating a fix for the anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}}. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:53:16,600] INFO Generating a fix for the anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}}. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
  [2021-04-15 18:53:17,146] INFO Goals used in proposal generation for UUID 5acbd9b5-70e1-43a8-8240-74a0ebf961f0 are already satisfied. (com.linkedin.kafka.cruisecontrol.KafkaCruiseControl)
  [2021-04-15 18:53:17,146] INFO Failed to generate proposals for execution (UUID: 5acbd9b5-70e1-43a8-8240-74a0ebf961f0 reason: Self healing for BROKER_FAILURE: {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}}). (com.linkedin.kafka.cruisecontrol.executor.Executor)

At '2021-04-15 20:00:36,773' the second timed execution for self healing is started.

[2021-04-15 20:00:36,773] INFO Fixing the anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}}. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
[2021-04-15 20:00:36,773] INFO Fixing the anomaly {Fixable broker failures detected: {Broker 4 failed at 15/04/2021 18:08:13}}. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
[2021-04-15 20:00:36,774] INFO [d7cd135a-7dc7-4e23-b18d-03e5f4e3a877] Self-healing started successfully. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)
[2021-04-15 20:00:36,774] INFO [d7cd135a-7dc7-4e23-b18d-03e5f4e3a877] Self-healing started successfully. (com.linkedin.kafka.cruisecontrol.detector.AnomalyDetectorManager)

[jiao-zhangS]

sorry, I figured out this. It was due to our cluster has some one-replica topics, so after first time's self healing, failed broker is still detected(because it's out of clusters and has assigned replicas). Then following up self-healing was triggered but did nothing. Sorry for bothering, okay to close this.