search

strimzi / strimzi-kafka-operator

Apache Kafka® running on Kubernetes
https://strimzi.io/
Apache License 2.0
4.8k stars 1.28k forks source link

[Question] When I reduced the number of replicas of the broker, I found that the status of Kafka changed to NotReady #4090

Closed AllenZMC closed 3 years ago

AllenZMC commented 3 years ago

When I reduced the number of replicas of the broker, I found that the status of Kafka changed to NotReady.

status:
  conditions:
  - lastTransitionTime: 2020-12-11T07:24:09+0000
    message: Pod kafka-test-cluster-kafka-0 is currently the controller and there
      are other pods still to roll
    reason: ForceableProblem
    status: "True"
    type: NotReady
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  creationTimestamp: '2020-11-30T02:20:37Z'
  generation: 36
  managedFields:
    - apiVersion: kafka.strimzi.io/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          'f:kafka':
            'f:authorization':
              .: {}
              'f:type': {}
            'f:config':
              'f:offsets.topic.replication.factor': {}
              'f:transaction.state.log.min.isr': {}
              'f:transaction.state.log.replication.factor': {}
            'f:listeners': {}
            'f:replicas': {}
      manager: Mozilla
      operation: Update
      time: '2020-12-11T08:07:18Z'
    - apiVersion: kafka.strimzi.io/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        'f:spec':
          'f:entityOperator':
            'f:topicOperator':
              'f:reconciliationIntervalSeconds': {}
              'f:topicMetadataMaxAttempts': {}
              'f:zookeeperSessionTimeoutSeconds': {}
            'f:userOperator':
              'f:reconciliationIntervalSeconds': {}
              'f:zookeeperSessionTimeoutSeconds': {}
        'f:status':
          .: {}
          'f:conditions': {}
          'f:listeners': {}
          'f:observedGeneration': {}
      manager: okhttp
      operation: Update
      time: '2020-12-11T08:12:05Z'
  name: kafka-test-cluster
  namespace: openshift-operators
  resourceVersion: '33246480'
  selfLink: >-
    /apis/kafka.strimzi.io/v1beta1/namespaces/openshift-operators/kafkas/kafka-test-cluster
  uid: 1530f196-09cc-43b6-bac7-84db5114f10c
spec:
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    authorization:
      type: simple
    config:
      log.message.format.version: '2.6'
      offsets.topic.replication.factor: 2
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 2
    listeners:
      - configuration:
          useServiceDnsDomain: true
        name: plain
        port: 9092
        tls: false
        type: internal
      - authentication:
          type: tls
        name: tls
        port: 9093
        tls: true
        type: internal
      - authentication:
          type: scram-sha-512
        name: external
        port: 9094
        tls: true
        type: nodeport
      - authentication:
          type: tls
        name: external2
        port: 9095
        tls: true
        type: nodeport
    replicas: 2
    storage:
      type: ephemeral
    version: 2.6.0
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral

Kafka operator : 0.20.0 ocp4.6

2020-12-11 07:22:20 DEBUG AbstractOperator:368 - Reconciliation #31(timer) Kafka(openshift-operators/testkafka): Lock lock::openshift-operators::Kafka::testkafka released
2020-12-11 07:22:25 WARN  AbstractOperator:377 - Reconciliation #32(timer) Kafka(openshift-operators/kafka-test-cluster): Failed to acquire lock lock::openshift-operators::Kafka::kafka-test-cluster within 10000ms.
2020-12-11 07:22:31 DEBUG KafkaRoller:270 - Reconciliation #28(watch) Kafka(openshift-operators/kafka-test-cluster): Considering restart of pod 0 after delay of 16000 MILLISECONDS
2020-12-11 07:22:31 DEBUG KafkaAssemblyOperator:3111 - Rolling update of openshift-operators/kafka-test-cluster-kafka: pod kafka-test-cluster-kafka-0 has strimzi.io/generation=0; sts has strimzi.io/generation=2
2020-12-11 07:22:31 DEBUG KafkaAssemblyOperator:3213 - Rolling update of openshift-operators/kafka-test-cluster-kafka: pod kafka-test-cluster-kafka-0 has strimzi.io/custom-listener-cert-thumbprints=; sts has strimzi.io/custom-listener-cert-thumbprints=
2020-12-11 07:22:31 DEBUG KafkaAssemblyOperator:3264 - Reconciliation #28(watch) Kafka(openshift-operators/kafka-test-cluster): Rolling pod kafka-test-cluster-kafka-0 due to [Pod has old generation]
2020-12-11 07:22:31 INFO  KafkaRoller:500 - Reconciliation #28(watch) Kafka(openshift-operators/kafka-test-cluster): Pod 0 needs to be restarted. Reason: [Pod has old generation]
2020-12-11 07:22:31 DEBUG KafkaRoller:687 - Reconciliation #28(watch) Kafka(openshift-operators/kafka-test-cluster): Creating AdminClient for kafka-test-cluster-kafka-0.kafka-test-cluster-kafka-brokers.openshift-operators.svc.cluster.local:9091
2020-12-11 07:22:32 DEBUG KafkaRoller:740 - Reconciliation #28(watch) Kafka(openshift-operators/kafka-test-cluster): Controller is 0
2020-12-11 07:22:32 DEBUG KafkaRoller:352 - Reconciliation #28(watch) Kafka(openshift-operators/kafka-test-cluster): Pod 0 is controller and there are other pods to roll
2020-12-11 07:22:32 INFO  KafkaRoller:294 - Reconciliation #28(watch) Kafka(openshift-operators/kafka-test-cluster): Could not roll pod 0 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod kafka-test-cluster-kafka-0 is currently the controller and there are other pods still to roll, retrying after at least 32000ms
2020-12-11 07:22:32 DEBUG KafkaRoller:270 - Reconciliation #28(watch) Kafka(openshift-operators/kafka-test-cluster): Considering restart of pod 1 after delay of 16000 MILLISECONDS
2020-12-11 07:22:32 DEBUG KafkaAssemblyOperator:3111 - Rolling update of openshift-operators/kafka-test-cluster-kafka: pod kafka-test-cluster-kafka-1 has strimzi.io/generation=0; sts has strimzi.io/generation=2
2020-12-11 07:22:32 DEBUG KafkaAssemblyOperator:3213 - Rolling update of openshift-operators/kafka-test-cluster-kafka: pod kafka-test-cluster-kafka-1 has strimzi.io/custom-listener-cert-thumbprints=; sts has strimzi.io/custom-listener-cert-thumbprints=
2020-12-11 07:22:32 DEBUG KafkaAssemblyOperator:3264 - Reconciliation #28(watch) Kafka(openshift-operators/kafka-test-cluster): Rolling pod kafka-test-cluster-kafka-1 due to [Pod has old generation]
2020-12-11 07:22:32 INFO  KafkaRoller:500 - Reconciliation #28(watch) Kafka(openshift-operators/kafka-test-cluster): Pod 1 needs to be restarted. Reason: [Pod has old generation]
2020-12-11 07:22:32 DEBUG KafkaRoller:687 - Reconciliation #28(watch) Kafka(openshift-operators/kafka-test-cluster): Creating AdminClient for kafka-test-cluster-kafka-1.kafka-test-cluster-kafka-brokers.openshift-operators.svc.cluster.local:9091
2020-12-11 07:22:32 DEBUG KafkaRoller:740 - Reconciliation #28(watch) Kafka(openshift-operators/kafka-test-cluster): Controller is 0
2020-12-11 07:22:32 DEBUG KafkaAvailability:58 - Determining whether broker 1 can be rolled
2020-12-11 07:22:32 DEBUG KafkaAvailability:205 - Got 5 topic names
2020-12-11 07:22:32 DEBUG KafkaAvailability:47 - Got 5 topic names
2020-12-11 07:22:32 DEBUG KafkaAvailability:191 - Got topic descriptions for 5 topics
2020-12-11 07:22:32 DEBUG KafkaAvailability:65 - Got 5 topic descriptions
2020-12-11 07:22:32 DEBUG KafkaAvailability:150 - Getting topic configs for 4 topics
2020-12-11 07:22:32 DEBUG KafkaAvailability:159 - Got topic configs for 4 topics
2020-12-11 07:22:32 DEBUG KafkaAvailability:97 - strimzi.cruisecontrol.partitionmetricsamples has min.insync.replicas=1.
2020-12-11 07:22:32 DEBUG KafkaAvailability:137 - strimzi.cruisecontrol.partitionmetricsamples/2 has ISR=[1], replicas=[2,1]
2020-12-11 07:22:32 INFO  KafkaAvailability:121 - strimzi.cruisecontrol.partitionmetricsamples/2 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 1 is restarted.
2020-12-11 07:22:32 DEBUG KafkaAvailability:82 - Restart pod 1 would remove it from ISR, stalling producers with acks=all
2020-12-11 07:22:32 DEBUG KafkaRoller:365 - Reconciliation #28(watch) Kafka(openshift-operators/kafka-test-cluster): Pod 1 cannot be rolled right now
2020-12-11 07:22:32 INFO  KafkaRoller:294 - Reconciliation #28(watch) Kafka(openshift-operators/kafka-test-cluster): Could not roll pod 1 due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod kafka-test-cluster-kafka-1 is currently not rollable, retrying after at least 32000ms
2020-12-11 07:38:13 INFO  KafkaRoller:287 - Reconciliation #1(watch) Kafka(openshift-operators/kafka-test-cluster): Could not roll pod 1, giving up after 10 attempts. Total delay between attempts 127750ms
io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod kafka-test-cluster-kafka-1 is currently not rollable
    at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:366) ~[io.strimzi.cluster-operator-0.20.0.jar:0.20.0]
    at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$5(KafkaRoller.java:272) ~[io.strimzi.cluster-operator-0.20.0.jar:0.20.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-11 07:38:13 ERROR AbstractOperator:238 - Reconciliation #1(watch) Kafka(openshift-operators/kafka-test-cluster): createOrUpdate failed
io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod kafka-test-cluster-kafka-0 is currently the controller and there are other pods still to roll
    at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:353) ~[io.strimzi.cluster-operator-0.20.0.jar:0.20.0]
    at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$5(KafkaRoller.java:272) ~[io.strimzi.cluster-operator-0.20.0.jar:0.20.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-11 07:38:13 DEBUG StatusDiff:39 - Ignoring Status diff {"op":"replace","path":"/conditions/0/lastTransitionTime","value":"2020-12-11T07:38:13+0000"}
2020-12-11 07:38:13 DEBUG AbstractOperator:318 - Reconciliation #1(watch) Kafka(openshift-operators/kafka-test-cluster): Status did not change
2020-12-11 07:38:13 DEBUG AbstractOperator:502 - Reconciliation #1(watch) Kafka(openshift-operators/kafka-test-cluster): Updated metric strimzi.resource.state[tag(kind=Kafka),tag(name=kafka-test-cluster),tag(resource-namespace=openshift-operators)] = 0
2020-12-11 07:38:13 WARN  AbstractOperator:470 - Reconciliation #1(watch) Kafka(openshift-operators/kafka-test-cluster): Failed to reconcile
io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod kafka-test-cluster-kafka-0 is currently the controller and there are other pods still to roll
    at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:353) ~[io.strimzi.cluster-operator-0.20.0.jar:0.20.0]
    at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$5(KafkaRoller.java:272) ~[io.strimzi.cluster-operator-0.20.0.jar:0.20.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-11 07:38:13 DEBUG AbstractOperator:368 - Reconciliation #1(watch) Kafka(openshift-operators/kafka-test-cluster): Lock lock::openshift-operators::Kafka::kafka-test-cluster released
AllenZMC commented 3 years ago 

2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:198 - Kafka Broker 2 Config Differs : {"op":"replace","path":"/cruise.control.metrics.reporter.ssl.truststore.password","value":"${CERTS_STORE_PASSWORD}"}
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:199 - Current Kafka Broker Config path cruise.control.metrics.reporter.ssl.truststore.password has value "null"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:200 - Desired Kafka Broker Config path cruise.control.metrics.reporter.ssl.truststore.password has value "${CERTS_STORE_PASSWORD}"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:198 - Kafka Broker 2 Config Differs : {"op":"replace","path":"/cruise.control.metrics.reporter.ssl.truststore.type","value":"PKCS12"}
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:199 - Current Kafka Broker Config path cruise.control.metrics.reporter.ssl.truststore.type has value "null"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:200 - Desired Kafka Broker Config path cruise.control.metrics.reporter.ssl.truststore.type has value "PKCS12"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:198 - Kafka Broker 2 Config Differs : {"op":"replace","path":"/cruise.control.metrics.topic","value":"strimzi.cruisecontrol.metrics"}
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:199 - Current Kafka Broker Config path cruise.control.metrics.topic has value "null"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:200 - Desired Kafka Broker Config path cruise.control.metrics.topic has value "strimzi.cruisecontrol.metrics"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:198 - Kafka Broker 2 Config Differs : {"op":"replace","path":"/cruise.control.metrics.topic.auto.create","value":"true"}
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:199 - Current Kafka Broker Config path cruise.control.metrics.topic.auto.create has value "null"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:200 - Desired Kafka Broker Config path cruise.control.metrics.topic.auto.create has value "true"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:198 - Kafka Broker 2 Config Differs : {"op":"replace","path":"/cruise.control.metrics.topic.num.partitions","value":"1"}
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:199 - Current Kafka Broker Config path cruise.control.metrics.topic.num.partitions has value "null"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:200 - Desired Kafka Broker Config path cruise.control.metrics.topic.num.partitions has value "1"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:198 - Kafka Broker 2 Config Differs : {"op":"replace","path":"/cruise.control.metrics.topic.replication.factor","value":"1"}
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:199 - Current Kafka Broker Config path cruise.control.metrics.topic.replication.factor has value "null"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:200 - Desired Kafka Broker Config path cruise.control.metrics.topic.replication.factor has value "1"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:198 - Kafka Broker 2 Config Differs : {"op":"replace","path":"/listener.name.replication-9091.ssl.keystore.password","value":"${CERTS_STORE_PASSWORD}"}
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:199 - Current Kafka Broker Config path listener.name.replication-9091.ssl.keystore.password has value "null"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:200 - Desired Kafka Broker Config path listener.name.replication-9091.ssl.keystore.password has value "${CERTS_STORE_PASSWORD}"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:198 - Kafka Broker 2 Config Differs : {"op":"replace","path":"/listener.name.replication-9091.ssl.truststore.password","value":"${CERTS_STORE_PASSWORD}"}
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:199 - Current Kafka Broker Config path listener.name.replication-9091.ssl.truststore.password has value "null"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:200 - Desired Kafka Broker Config path listener.name.replication-9091.ssl.truststore.password has value "${CERTS_STORE_PASSWORD}"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:198 - Kafka Broker 2 Config Differs : {"op":"replace","path":"/listener.name.tls-9093.ssl.keystore.password","value":"${CERTS_STORE_PASSWORD}"}
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:199 - Current Kafka Broker Config path listener.name.tls-9093.ssl.keystore.password has value "null"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:200 - Desired Kafka Broker Config path listener.name.tls-9093.ssl.keystore.password has value "${CERTS_STORE_PASSWORD}"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:198 - Kafka Broker 2 Config Differs : {"op":"replace","path":"/listener.name.tls-9093.ssl.truststore.password","value":"${CERTS_STORE_PASSWORD}"}
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:199 - Current Kafka Broker Config path listener.name.tls-9093.ssl.truststore.password has value "null"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:200 - Desired Kafka Broker Config path listener.name.tls-9093.ssl.truststore.password has value "${CERTS_STORE_PASSWORD}"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:198 - Kafka Broker 2 Config Differs : {"op":"replace","path":"/super.users","value":"User:CN=testkafka-kafka,O=io.strimzi;User:CN=testkafka-entity-operator,O=io.strimzi;User:CN=testkafka-kafka-exporter,O=io.strimzi;User:CN=testkafka-cruise-control,O=io.strimzi;User:CN=cluster-operator,O=io.strimzi"}
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:199 - Current Kafka Broker Config path super.users has value "null"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:200 - Desired Kafka Broker Config path super.users has value "User:CN=testkafka-kafka,O=io.strimzi;User:CN=testkafka-entity-operator,O=io.strimzi;User:CN=testkafka-kafka-exporter,O=io.strimzi;User:CN=testkafka-cruise-control,O=io.strimzi;User:CN=cluster-operator,O=io.strimzi"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:198 - Kafka Broker 2 Config Differs : {"op":"replace","path":"/zookeeper.ssl.keystore.password","value":"${CERTS_STORE_PASSWORD}"}
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:199 - Current Kafka Broker Config path zookeeper.ssl.keystore.password has value "null"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:200 - Desired Kafka Broker Config path zookeeper.ssl.keystore.password has value "${CERTS_STORE_PASSWORD}"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:198 - Kafka Broker 2 Config Differs : {"op":"replace","path":"/zookeeper.ssl.truststore.password","value":"${CERTS_STORE_PASSWORD}"}
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:199 - Current Kafka Broker Config path zookeeper.ssl.truststore.password has value "null"
2020-12-11 07:51:53 DEBUG KafkaBrokerConfigurationDiff:200 - Desired Kafka Broker Config path zookeeper.ssl.truststore.password has value "${CERTS_STORE_PASSWORD}"
2020-12-11 07:51:53 DEBUG KafkaBrokerLoggingConfigurationDiff:146 - Skipping log4j line: log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
2020-12-11 07:51:53 DEBUG KafkaBrokerLoggingConfigurationDiff:146 - Skipping log4j line: log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
2020-12-11 07:51:53 DEBUG KafkaBrokerLoggingConfigurationDiff:146 - Skipping log4j line: log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} %p %m (%c) [%t]%n
2020-12-11 07:51:53 DEBUG KafkaBrokerLoggingConfigurationDiff:122 - Treating the line as ENV var declaration: kafka.root.logger.level=INFO
2020-12-11 07:51:53 DEBUG KafkaRoller:373 - Reconciliation #35(timer) Kafka(openshift-operators/testkafka): Pod 2 does not need to be restarted
2020-12-11 07:51:53 DEBUG KafkaRoller:374 - Reconciliation #35(timer) Kafka(openshift-operators/testkafka): Waiting for non-restarted pod 2 to become ready
2020-12-11 07:51:53 DEBUG Util:93 - Waiting for Pods resource testkafka-kafka-2 in namespace openshift-operators to get ready
2020-12-11 07:51:53 DEBUG SecretOperator:102 - Secret openshift-marketplace/my-cluster-entity-operator-certs already exists, patching it
2020-12-11 07:51:53 DEBUG Util:115 - Pods resource testkafka-kafka-2 in namespace openshift-operators is ready
2020-12-11 07:51:53 DEBUG KafkaRoller:376 - Reconciliation #35(timer) Kafka(openshift-operators/testkafka): Pod 2 is now ready
2020-12-11 07:51:53 DEBUG SecretOperator:168 - Secret my-cluster-entity-operator-certs in namespace openshift-marketplace has been patched
2020-12-11 07:51:53 DEBUG KafkaAssemblyOperator:2614 - Reconciliation #35(timer) Kafka(openshift-operators/testkafka): Checking readiness of pod testkafka-kafka-0.
2020-12-11 07:51:53 DEBUG Util:93 - Waiting for Pods resource testkafka-kafka-0 in namespace openshift-operators to get ready
2020-12-11 07:51:53 DEBUG KafkaAssemblyOperator:2614 - Reconciliation #35(timer) Kafka(openshift-operators/testkafka): Checking readiness of pod testkafka-kafka-1.
2020-12-11 07:51:53 DEBUG Util:93 - Waiting for Pods resource testkafka-kafka-1 in namespace openshift-operators to get ready
2020-12-11 07:51:53 DEBUG KafkaAssemblyOperator:2614 - Reconciliation #35(timer) Kafka(openshift-operators/testkafka): Checking readiness of pod testkafka-kafka-2.
2020-12-11 07:51:53 DEBUG Util:93 - Waiting for Pods resource testkafka-kafka-2 in namespace openshift-operators to get ready
2020-12-11 07:51:53 DEBUG DeploymentOperator:102 - Deployment openshift-marketplace/my-cluster-entity-operator already exists, patching it
2020-12-11 07:51:53 DEBUG Util:115 - Pods resource testkafka-kafka-0 in namespace openshift-operators is ready
2020-12-11 07:51:53 DEBUG DeploymentOperator:168 - Deployment my-cluster-entity-operator in namespace openshift-marketplace has been patched
2020-12-11 07:51:53 DEBUG Util:115 - Pods resource testkafka-kafka-1 in namespace openshift-operators is ready
2020-12-11 07:51:53 DEBUG Util:93 - Waiting for Deployment resource my-cluster-entity-operator in namespace openshift-marketplace to get observed
2020-12-11 07:51:53 DEBUG Util:115 - Pods resource testkafka-kafka-2 in namespace openshift-operators is ready
2020-12-11 07:51:53 DEBUG Util:93 - Waiting for Endpoints resource testkafka-kafka-bootstrap in namespace openshift-operators to get ready
2020-12-11 07:51:53 DEBUG Util:115 - Deployment resource my-cluster-entity-operator in namespace openshift-marketplace is observed
2020-12-11 07:51:53 DEBUG Util:93 - Waiting for Deployment resource my-cluster-entity-operator in namespace openshift-marketplace to get ready
2020-12-11 07:51:53 DEBUG Util:115 - Endpoints resource testkafka-kafka-bootstrap in namespace openshift-operators is ready
2020-12-11 07:51:53 DEBUG Util:93 - Waiting for Endpoints resource testkafka-kafka-brokers in namespace openshift-operators to get ready
2020-12-11 07:51:53 DEBUG Util:115 - Deployment resource my-cluster-entity-operator in namespace openshift-marketplace is ready
2020-12-11 07:51:53 DEBUG Util:115 - Endpoints resource testkafka-kafka-brokers in namespace openshift-operators is ready
2020-12-11 07:51:53 DEBUG NetworkPolicyOperator:111 - NetworkPolicy openshift-marketplace/my-cluster-network-policy-cruise-control does not exist, noop
2020-12-11 07:51:53 DEBUG KafkaAssemblyOperator:3371 - Adding certificate to status for listener: tls
2020-12-11 07:51:53 DEBUG AbstractModel:472 - logging is not set, using default loggers
2020-12-11 07:51:53 DEBUG ServiceAccountOperator:111 - ServiceAccount openshift-marketplace/my-cluster-cruise-control does not exist, noop
2020-12-11 07:51:53 DEBUG AbstractModel:472 - logging is not set, using default loggers
2020-12-11 07:51:53 DEBUG ConfigMapOperator:111 - ConfigMap openshift-marketplace/my-cluster-cruise-control-config does not exist, noop
2020-12-11 07:51:53 DEBUG ServiceAccountOperator:102 - ServiceAccount openshift-operators/testkafka-entity-operator already exists, patching it
2020-12-11 07:51:53 DEBUG ServiceAccountOperator:35 - ServiceAccount testkafka-entity-operator in namespace openshift-operators has not been patched: patching service accounts generates new tokens which should be avoided.
2020-12-11 07:51:53 DEBUG RoleBindingOperator:102 - RoleBinding openshift-operators/strimzi-testkafka-entity-topic-operator already exists, patching it
2020-12-11 07:51:53 DEBUG SecretOperator:111 - Secret openshift-marketplace/my-cluster-cruise-control-certs does not exist, noop
2020-12-11 07:51:53 DEBUG RoleBindingOperator:168 - RoleBinding strimzi-testkafka-entity-topic-operator in namespace openshift-operators has been patched
2020-12-11 07:51:53 DEBUG DeploymentOperator:111 - Deployment openshift-marketplace/my-cluster-cruise-control does not exist, noop
2020-12-11 07:51:53 DEBUG RoleBindingOperator:102 - RoleBinding openshift-operators/strimzi-testkafka-entity-user-operator already exists, patching it
2020-12-11 07:51:53 DEBUG ServiceOperator:111 - Service openshift-marketplace/my-cluster-cruise-control does not exist, noop
2020-12-11 07:51:53 DEBUG ServiceAccountOperator:111 - ServiceAccount openshift-marketplace/my-cluster-kafka-exporter does not exist, noop
2020-12-11 07:51:53 DEBUG RoleBindingOperator:168 - RoleBinding strimzi-testkafka-entity-user-operator in namespace openshift-operators has been patched
2020-12-11 07:51:53 DEBUG ConfigMapOperator:102 - ConfigMap openshift-operators/testkafka-entity-topic-operator-config already exists, patching it
2020-12-11 07:51:53 DEBUG ConfigMapOperator:47 - ConfigMap testkafka-entity-topic-operator-config in namespace openshift-operators has not been patched because resources are equal
2020-12-11 07:51:53 DEBUG ConfigMapOperator:102 - ConfigMap openshift-operators/testkafka-entity-user-operator-config already exists, patching it
2020-12-11 07:51:53 DEBUG ConfigMapOperator:47 - ConfigMap testkafka-entity-user-operator-config in namespace openshift-operators has not been patched because resources are equal
2020-12-11 07:51:53 DEBUG SecretOperator:111 - Secret openshift-marketplace/my-cluster-kafka-exporter-certs does not exist, noop
2020-12-11 07:51:53 DEBUG DeploymentOperator:111 - Deployment openshift-marketplace/my-cluster-kafka-exporter does not exist, noop
2020-12-11 07:51:53 DEBUG SecretOperator:102 - Secret openshift-operators/testkafka-entity-operator-certs already exists, patching it
2020-12-11 07:51:53 DEBUG ServiceAccountOperator:111 - ServiceAccount openshift-marketplace/my-cluster-kafka-jmx-trans does not exist, noop
2020-12-11 07:51:53 DEBUG ConfigMapOperator:111 - ConfigMap openshift-marketplace/my-cluster-jmxtrans-config does not exist, noop
2020-12-11 07:51:53 DEBUG SecretOperator:168 - Secret testkafka-entity-operator-certs in namespace openshift-operators has been patched
2020-12-11 07:51:53 DEBUG DeploymentOperator:111 - Deployment openshift-marketplace/my-cluster-kafka-jmx-trans does not exist, noop
2020-12-11 07:51:53 DEBUG StatusDiff:39 - Ignoring Status diff {"op":"replace","path":"/conditions/0/lastTransitionTime","value":"2020-12-11T07:51:53+0000"}
2020-12-11 07:51:53 DEBUG AbstractOperator:318 - Reconciliation #34(timer) Kafka(openshift-marketplace/my-cluster): Status did not change
2020-12-11 07:51:53 DEBUG DeploymentOperator:102 - Deployment openshift-operators/testkafka-entity-operator already exists, patching it
2020-12-11 07:51:53 DEBUG AbstractOperator:502 - Reconciliation #34(timer) Kafka(openshift-marketplace/my-cluster): Updated metric strimzi.resource.state[tag(kind=Kafka),tag(name=my-cluster),tag(resource-namespace=openshift-marketplace)] = 1
2020-12-11 07:51:53 INFO  AbstractOperator:455 - Reconciliation #34(timer) Kafka(openshift-marketplace/my-cluster): reconciled
2020-12-11 07:51:53 DEBUG AbstractOperator:368 - Reconciliation #34(timer) Kafka(openshift-marketplace/my-cluster): Lock lock::openshift-marketplace::Kafka::my-cluster released
2020-12-11 07:51:53 DEBUG DeploymentOperator:168 - Deployment testkafka-entity-operator in namespace openshift-operators has been patched
2020-12-11 07:51:53 DEBUG Util:93 - Waiting for Deployment resource testkafka-entity-operator in namespace openshift-operators to get observed
2020-12-11 07:51:53 DEBUG Util:115 - Deployment resource testkafka-entity-operator in namespace openshift-operators is observed
2020-12-11 07:51:53 DEBUG Util:93 - Waiting for Deployment resource testkafka-entity-operator in namespace openshift-operators to get ready
2020-12-11 07:51:53 DEBUG Util:115 - Deployment resource testkafka-entity-operator in namespace openshift-operators is ready
2020-12-11 07:51:53 DEBUG NetworkPolicyOperator:102 - NetworkPolicy openshift-operators/testkafka-network-policy-cruise-control already exists, patching it
2020-12-11 07:51:53 DEBUG NetworkPolicyOperator:168 - NetworkPolicy testkafka-network-policy-cruise-control in namespace openshift-operators has been patched
2020-12-11 07:51:53 DEBUG ServiceAccountOperator:102 - ServiceAccount openshift-operators/testkafka-cruise-control already exists, patching it
2020-12-11 07:51:53 DEBUG ServiceAccountOperator:35 - ServiceAccount testkafka-cruise-control in namespace openshift-operators has not been patched: patching service accounts generates new tokens which should be avoided.
2020-12-11 07:51:53 DEBUG ConfigMapOperator:102 - ConfigMap openshift-operators/testkafka-cruise-control-config already exists, patching it
2020-12-11 07:51:53 DEBUG ConfigMapOperator:47 - ConfigMap testkafka-cruise-control-config in namespace openshift-operators has not been patched because resources are equal
2020-12-11 07:51:53 DEBUG SecretOperator:102 - Secret openshift-operators/testkafka-cruise-control-certs already exists, patching it
2020-12-11 07:51:53 DEBUG SecretOperator:168 - Secret testkafka-cruise-control-certs in namespace openshift-operators has been patched
2020-12-11 07:51:53 DEBUG DeploymentOperator:102 - Deployment openshift-operators/testkafka-cruise-control already exists, patching it
2020-12-11 07:51:53 DEBUG DeploymentOperator:168 - Deployment testkafka-cruise-control in namespace openshift-operators has been patched
2020-12-11 07:51:53 DEBUG ServiceOperator:102 - Service openshift-operators/testkafka-cruise-control already exists, patching it
2020-12-11 07:51:53 DEBUG ServiceOperator:168 - Service testkafka-cruise-control in namespace openshift-operators has been patched
2020-12-11 07:51:53 DEBUG Util:93 - Waiting for Deployment resource testkafka-cruise-control in namespace openshift-operators to get observed
2020-12-11 07:51:53 DEBUG Util:115 - Deployment resource testkafka-cruise-control in namespace openshift-operators is observed
2020-12-11 07:51:53 DEBUG Util:93 - Waiting for Deployment resource testkafka-cruise-control in namespace openshift-operators to get ready
2020-12-11 07:51:53 DEBUG Util:115 - Deployment resource testkafka-cruise-control in namespace openshift-operators is ready
2020-12-11 07:51:53 DEBUG ServiceAccountOperator:111 - ServiceAccount openshift-operators/testkafka-kafka-exporter does not exist, noop
2020-12-11 07:51:53 DEBUG SecretOperator:111 - Secret openshift-operators/testkafka-kafka-exporter-certs does not exist, noop
2020-12-11 07:51:53 DEBUG DeploymentOperator:111 - Deployment openshift-operators/testkafka-kafka-exporter does not exist, noop
2020-12-11 07:51:53 DEBUG ServiceAccountOperator:111 - ServiceAccount openshift-operators/testkafka-kafka-jmx-trans does not exist, noop
2020-12-11 07:51:53 DEBUG ConfigMapOperator:111 - ConfigMap openshift-operators/testkafka-jmxtrans-config does not exist, noop
2020-12-11 07:51:53 DEBUG DeploymentOperator:111 - Deployment openshift-operators/testkafka-kafka-jmx-trans does not exist, noop
2020-12-11 07:51:53 DEBUG StatusDiff:39 - Ignoring Status diff {"op":"replace","path":"/conditions/0/lastTransitionTime","value":"2020-12-11T07:51:53+0000"}
2020-12-11 07:51:53 DEBUG AbstractOperator:318 - Reconciliation #35(timer) Kafka(openshift-operators/testkafka): Status did not change
2020-12-11 07:51:53 DEBUG AbstractOperator:502 - Reconciliation #35(timer) Kafka(openshift-operators/testkafka): Updated metric strimzi.resource.state[tag(kind=Kafka),tag(name=testkafka),tag(resource-namespace=openshift-operators)] = 1
2020-12-11 07:51:53 INFO  AbstractOperator:455 - Reconciliation #35(timer) Kafka(openshift-operators/testkafka): reconciled
2020-12-11 07:51:53 DEBUG AbstractOperator:368 - Reconciliation #35(timer) Kafka(openshift-operators/testkafka): Lock lock::openshift-operators::Kafka::testkafka released
2020-12-11 07:52:00 WARN  AbstractOperator:377 - Reconciliation #36(timer) Kafka(openshift-operators/kafka-test-cluster): Failed to acquire lock lock::openshift-operators::Kafka::kafka-test-cluster within 10000ms.
2020-12-11 07:52:04 DEBUG KafkaRoller:270 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Considering restart of pod 0 after delay of 64000 MILLISECONDS
2020-12-11 07:52:04 DEBUG KafkaAssemblyOperator:3111 - Rolling update of openshift-operators/kafka-test-cluster-kafka: pod kafka-test-cluster-kafka-0 has strimzi.io/generation=0; sts has strimzi.io/generation=3
2020-12-11 07:52:04 DEBUG KafkaAssemblyOperator:3213 - Rolling update of openshift-operators/kafka-test-cluster-kafka: pod kafka-test-cluster-kafka-0 has strimzi.io/custom-listener-cert-thumbprints=; sts has strimzi.io/custom-listener-cert-thumbprints=
2020-12-11 07:52:04 DEBUG KafkaAssemblyOperator:3264 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Rolling pod kafka-test-cluster-kafka-0 due to [Pod has old generation]
2020-12-11 07:52:04 INFO  KafkaRoller:500 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Pod 0 needs to be restarted. Reason: [Pod has old generation]
2020-12-11 07:52:04 DEBUG KafkaRoller:687 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Creating AdminClient for kafka-test-cluster-kafka-0.kafka-test-cluster-kafka-brokers.openshift-operators.svc.cluster.local:9091
2020-12-11 07:52:04 DEBUG KafkaRoller:740 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Controller is 0
2020-12-11 07:52:04 DEBUG KafkaRoller:352 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Pod 0 is controller and there are other pods to roll
2020-12-11 07:52:04 DEBUG KafkaAvailability:58 - Determining whether broker 0 can be rolled
2020-12-11 07:52:04 DEBUG KafkaAvailability:205 - Got 5 topic names
2020-12-11 07:52:04 DEBUG KafkaAvailability:47 - Got 5 topic names
2020-12-11 07:52:04 DEBUG KafkaAvailability:191 - Got topic descriptions for 5 topics
2020-12-11 07:52:04 DEBUG KafkaAvailability:65 - Got 5 topic descriptions
2020-12-11 07:52:04 DEBUG KafkaAvailability:150 - Getting topic configs for 4 topics
2020-12-11 07:52:04 DEBUG KafkaAvailability:159 - Got topic configs for 4 topics
2020-12-11 07:52:04 DEBUG KafkaAvailability:97 - strimzi.cruisecontrol.partitionmetricsamples has min.insync.replicas=1.
2020-12-11 07:52:04 DEBUG KafkaAvailability:137 - strimzi.cruisecontrol.partitionmetricsamples/1 has ISR=[0], replicas=[0,2]
2020-12-11 07:52:04 INFO  KafkaAvailability:121 - strimzi.cruisecontrol.partitionmetricsamples/1 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 0 is restarted.
2020-12-11 07:52:04 DEBUG KafkaAvailability:82 - Restart pod 0 would remove it from ISR, stalling producers with acks=all
2020-12-11 07:52:04 WARN  KafkaRoller:384 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Pod kafka-test-cluster-kafka-0 can't be safely force-rolled; original error:
2020-12-11 07:52:04 INFO  KafkaRoller:287 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Could not roll pod 0, giving up after 10 attempts. Total delay between attempts 127750ms
io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod kafka-test-cluster-kafka-0 is currently the controller and there are other pods still to roll
    at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:353) ~[io.strimzi.cluster-operator-0.20.0.jar:0.20.0]
    at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$5(KafkaRoller.java:272) ~[io.strimzi.cluster-operator-0.20.0.jar:0.20.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-11 07:52:04 DEBUG KafkaRoller:270 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Considering restart of pod 1 after delay of 64000 MILLISECONDS
2020-12-11 07:52:04 DEBUG KafkaAssemblyOperator:3111 - Rolling update of openshift-operators/kafka-test-cluster-kafka: pod kafka-test-cluster-kafka-1 has strimzi.io/generation=0; sts has strimzi.io/generation=3
2020-12-11 07:52:04 DEBUG KafkaAssemblyOperator:3213 - Rolling update of openshift-operators/kafka-test-cluster-kafka: pod kafka-test-cluster-kafka-1 has strimzi.io/custom-listener-cert-thumbprints=; sts has strimzi.io/custom-listener-cert-thumbprints=
2020-12-11 07:52:04 DEBUG KafkaAssemblyOperator:3264 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Rolling pod kafka-test-cluster-kafka-1 due to [Pod has old generation]
2020-12-11 07:52:04 INFO  KafkaRoller:500 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Pod 1 needs to be restarted. Reason: [Pod has old generation]
2020-12-11 07:52:04 DEBUG KafkaRoller:687 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Creating AdminClient for kafka-test-cluster-kafka-1.kafka-test-cluster-kafka-brokers.openshift-operators.svc.cluster.local:9091
2020-12-11 07:52:05 DEBUG KafkaRoller:740 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Controller is 0
2020-12-11 07:52:05 DEBUG KafkaAvailability:58 - Determining whether broker 1 can be rolled
2020-12-11 07:52:05 DEBUG KafkaAvailability:205 - Got 5 topic names
2020-12-11 07:52:05 DEBUG KafkaAvailability:47 - Got 5 topic names
2020-12-11 07:52:05 DEBUG KafkaAvailability:191 - Got topic descriptions for 5 topics
2020-12-11 07:52:05 DEBUG KafkaAvailability:65 - Got 5 topic descriptions
2020-12-11 07:52:05 DEBUG KafkaAvailability:150 - Getting topic configs for 4 topics
2020-12-11 07:52:05 DEBUG KafkaAvailability:159 - Got topic configs for 4 topics
2020-12-11 07:52:05 DEBUG KafkaAvailability:97 - strimzi.cruisecontrol.partitionmetricsamples has min.insync.replicas=1.
2020-12-11 07:52:05 DEBUG KafkaAvailability:137 - strimzi.cruisecontrol.partitionmetricsamples/2 has ISR=[1], replicas=[2,1]
2020-12-11 07:52:05 INFO  KafkaAvailability:121 - strimzi.cruisecontrol.partitionmetricsamples/2 will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 1 is restarted.
2020-12-11 07:52:05 DEBUG KafkaAvailability:82 - Restart pod 1 would remove it from ISR, stalling producers with acks=all
2020-12-11 07:52:05 DEBUG KafkaRoller:365 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Pod 1 cannot be rolled right now
2020-12-11 07:52:05 INFO  KafkaRoller:287 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Could not roll pod 1, giving up after 10 attempts. Total delay between attempts 127750ms
io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod kafka-test-cluster-kafka-1 is currently not rollable
    at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:366) ~[io.strimzi.cluster-operator-0.20.0.jar:0.20.0]
    at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$5(KafkaRoller.java:272) ~[io.strimzi.cluster-operator-0.20.0.jar:0.20.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-11 07:52:05 ERROR AbstractOperator:238 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): createOrUpdate failed
io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod kafka-test-cluster-kafka-0 is currently the controller and there are other pods still to roll
    at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:353) ~[io.strimzi.cluster-operator-0.20.0.jar:0.20.0]
    at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$5(KafkaRoller.java:272) ~[io.strimzi.cluster-operator-0.20.0.jar:0.20.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-11 07:52:05 DEBUG StatusDiff:39 - Ignoring Status diff {"op":"replace","path":"/conditions/0/lastTransitionTime","value":"2020-12-11T07:52:05+0000"}
2020-12-11 07:52:05 DEBUG AbstractOperator:318 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Status did not change
2020-12-11 07:52:05 DEBUG AbstractOperator:502 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Updated metric strimzi.resource.state[tag(kind=Kafka),tag(name=kafka-test-cluster),tag(resource-namespace=openshift-operators)] = 0
2020-12-11 07:52:05 WARN  AbstractOperator:470 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Failed to reconcile
io.strimzi.operator.cluster.operator.resource.KafkaRoller$ForceableProblem: Pod kafka-test-cluster-kafka-0 is currently the controller and there are other pods still to roll
    at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:353) ~[io.strimzi.cluster-operator-0.20.0.jar:0.20.0]
    at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$5(KafkaRoller.java:272) ~[io.strimzi.cluster-operator-0.20.0.jar:0.20.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
2020-12-11 07:52:05 DEBUG AbstractOperator:368 - Reconciliation #33(timer) Kafka(openshift-operators/kafka-test-cluster): Lock lock::openshift-operators::Kafka::kafka-test-cluster released
scholzj commented 3 years ago

Before you scale-down the Kafka cluster, you have to move all the topics from the brokers which will be deleted (they are always deleted from the end => so those with highest index number first). Strimzi currently does not move the pods automatically (one day it probably will, just not yet implemented). From the log I guess it looks like you have some replicas on the deleted brokers and they do not allow the operator to roll the other pods without making the topics underreplicated.

AllenZMC commented 3 years ago

If I want to reduce the size of broker, then I have to move all the topics from the brokers which will be deleted ? And how to move all the topics from the brokers which will be deleted?

AllenZMC commented 3 years ago

Why roll Broker-0, I changed the replicas from 3 to 2, should only delete Broker-3, why restart Broker-0 ?

scholzj commented 3 years ago

It is a longer story, but you need to restart all brokers to scale up or down right now. So that is expected. And yes, Kafka does not automatically move the topics before scale down or after scale up. And as I said - as of now Strimzi does not do it either. So you have to do it your self. After scale-up you can for sure use Cruise Control and KafkaRebalance resource for it.