[Bug] mm2 recreate doesn't seem to work

[carlobongiovanni]

Describe the bug Hi, I'm playing with mm2 and on the first install of it, it worked correctly, the resources are created and all looks fine. after that I deleted the mm2 and reinstalled it and according to the operator there is nothing to be done, but the mm2 deployment is not created anymore.

I'm using the release 0.17.0-rc2

2020-03-18 15:47:37 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace *...
2020-03-18 15:47:37 INFO  AbstractOperator:122 - Reconciliation #367(timer) Kafka(nonprod-seller-kfk/seller): Kafka seller should be created or updated
2020-03-18 15:47:37 INFO  AbstractOperator:122 - Reconciliation #368(timer) KafkaMirrorMaker2(nonprod-seller-kfk/seller-mm2-cluster): KafkaMirrorMaker2 seller-mm2-cluster should be created or updated
2020-03-18 15:47:37 INFO  AbstractOperator:262 - Reconciliation #368(timer) KafkaMirrorMaker2(nonprod-seller-kfk/seller-mm2-cluster): reconciled
2020-03-18 15:47:38 INFO  AbstractOperator:262 - Reconciliation #367(timer) Kafka(nonprod-seller-kfk/seller): reconciled

the mm2 cluster is setup to use as source a non-strimzi kafka cluster on version 2.0, and as target cluster a strimzi kafka 2.4.0

I have tried to delete the topics left after the installation of mm2 on the destination cluster, and I have also tried to delete completely the destination cluster, but it seems it makes no difference.

I have also tried installing in a separate namespace, same results.

I haven't tried to delete the topics created in the source cluster.

Please note that on my setup I have also done the upgrade of the strimzi operator, the first installed version was 0.14.0 and I have upgraded it up to the latest one.

To Reproduce Steps to reproduce the behavior:

1. install a cluster 2.4.0
2. install mm2
3. delete mm2
4. try to install mm2 again

Expected behavior The mm2 should be recreated.

Environment (please complete the following information):

• Strimzi version: [0.17.0-rc2]
• Installation method: [YAML files, Helm chart]
• Kubernetes cluster: [GKE]
• Infrastructure: [Google Cloud]

YAML files and logs

 apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaMirrorMaker2
metadata:
  name: seller-mm2-cluster
spec:
  version: 2.4.0
  replicas: 2
  # target
  connectCluster: {{ .Values.clusterName }}
  clusters:
  - alias: "ecg-nonprod"
    bootstrapServers: source-lb:9092
  - alias: "euwe-seller-nonprod"
    bootstrapServers: target-lb:9094
    authentication:
      type: scram-sha-512
      username: mm2-connect
      passwordSecret:
        secretName: mm2-connect
        password: password
    config:
      config.storage.replication.factor: 2
      offset.storage.replication.factor: 2
      status.storage.replication.factor: 2
    tls:
      trustedCertificates:
      - certificate: ca.crt
        secretName: {{ .Values.clusterName }}-cluster-ca-cert
  mirrors:
  - sourceCluster: "ecg-nonprod"
    targetCluster: "euwe-seller-nonprod"
    sourceConnector:
      config:
        replication.factor: 1
        offset-syncs.topic.replication.factor: 1
        sync.topic.acls.enabled: "false"
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 1
    checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 1
    topicsPattern: "^seller.*|^move-seller.*"
    groupsPattern: ".*"

[scholzj]

Could you share debug logs from the Cluster Operator? You can enable them by setting the environment variable STRIMZI_LOG_LEVEL in the cluster operator to DEBUG. That should tell us more.

Also, if you do the steps 3 and 4 very quickly after each other, it might sometimes take some more time to get created because of how Kubernetes garbage collection which does the deletion works.

[scholzj]

Als, if you use different versions of the Strimzi operator (or any operator) in the same cluster you need to be careful not to override the cluster wide resources (Cluster Roles / Custom Resource Definition) from the the newer version with some odler ones and make sure the ClusterRoleBindings for both versions are properly configured. These are cluster wide resources which can exist only once, so this could also cause some problems.

[scholzj]

Actually, in your case the operator is listening to all namespaces ... in that case it should be always the only one in the cluster. Maybe I misunderstood and you upgraded from 0.14 to the 0.17.0-rc2?

[carlobongiovanni]

Could you share debug logs from the Cluster Operator? You can enable them by setting the environment variable STRIMZI_LOG_LEVEL in the cluster operator to DEBUG. That should tell us more.

2020-03-19 08:35:36 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace *...
2020-03-19 08:35:36 DEBUG AbstractOperator:173 - Reconciliation #35(timer) Kafka(nonprod-seller-kfk/seller): Lock lock::nonprod-seller-kfk::Kafka::seller acquired
2020-03-19 08:35:36 INFO  AbstractOperator:122 - Reconciliation #35(timer) Kafka(nonprod-seller-kfk/seller): Kafka seller should be created or updated
2020-03-19 08:35:36 DEBUG AbstractOperator:173 - Reconciliation #36(timer) KafkaMirrorMaker2(nonprod-seller-kfk/seller-mm2-cluster): Lock lock::nonprod-seller-kfk::KafkaMirrorMaker2::seller-mm2-cluster acquired
2020-03-19 08:35:36 INFO  AbstractOperator:122 - Reconciliation #36(timer) KafkaMirrorMaker2(nonprod-seller-kfk/seller-mm2-cluster): KafkaMirrorMaker2 seller-mm2-cluster should be created or updated
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:515 - Reconciliation #35(timer) Kafka(nonprod-seller-kfk/seller): Status is already set. No need to set initial status
2020-03-19 08:35:36 DEBUG StatusDiff:37 - Ignoring Status diff {"op":"replace","path":"/conditions/0/lastTransitionTime","value":"2020-03-19T08:35:36.676Z"}
2020-03-19 08:35:36 DEBUG AbstractConnectOperator:468 - Reconciliation #36(timer) KafkaMirrorMaker2(nonprod-seller-kfk/seller-mm2-cluster): Status did not change
2020-03-19 08:35:36 INFO  AbstractOperator:262 - Reconciliation #36(timer) KafkaMirrorMaker2(nonprod-seller-kfk/seller-mm2-cluster): reconciled
2020-03-19 08:35:36 DEBUG AbstractOperator:184 - Reconciliation #36(timer) KafkaMirrorMaker2(nonprod-seller-kfk/seller-mm2-cluster): Lock lock::nonprod-seller-kfk::KafkaMirrorMaker2::seller-mm2-cluster released
2020-03-19 08:35:36 DEBUG Ca:649 - clients-ca: The CA certificate in secret seller-clients-ca-cert already exists and does not need renewing
2020-03-19 08:35:36 DEBUG Ca:508 - clients-ca renewalType NOOP
2020-03-19 08:35:36 DEBUG SecretOperator:102 - Secret nonprod-seller-kfk/seller-clients-ca-cert already exists, patching it
2020-03-19 08:35:36 DEBUG SecretOperator:168 - Secret seller-clients-ca-cert in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:36 DEBUG SecretOperator:102 - Secret nonprod-seller-kfk/seller-clients-ca already exists, patching it
2020-03-19 08:35:36 DEBUG SecretOperator:168 - Secret seller-clients-ca in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:36 DEBUG SecretOperator:102 - Secret nonprod-seller-kfk/seller-cluster-operator-certs already exists, patching it
2020-03-19 08:35:36 DEBUG SecretOperator:168 - Secret seller-cluster-operator-certs in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:2341 - Reconciliation #35(timer) Kafka(nonprod-seller-kfk/seller): Considering manual cleaning of Pods for StatefulSet seller-zookeeper
2020-03-19 08:35:36 DEBUG NetworkPolicyOperator:102 - NetworkPolicy nonprod-seller-kfk/seller-network-policy-zookeeper already exists, patching it
2020-03-19 08:35:36 DEBUG NetworkPolicyOperator:168 - NetworkPolicy seller-network-policy-zookeeper in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:835 - STS seller-kafka currently has Kafka version KafkaVersion{version='2.4.0', protocolVersion='2.4', messageVersion='2.4', zookeeperVersion='3.5.6', isDefault=true, unsupportedFeatures='null'}
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:845 - STS seller-kafka is moving from Kafka version KafkaVersion{version='2.4.0', protocolVersion='2.4', messageVersion='2.4', zookeeperVersion='3.5.6', isDefault=true, unsupportedFeatures='null'}
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:855 - STS seller-kafka is moving to Kafka version KafkaVersion{version='2.4.0', protocolVersion='2.4', messageVersion='2.4', zookeeperVersion='3.5.6', isDefault=true, unsupportedFeatures='null'}
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:859 - Kafka version change: Kafka version=KafkaVersion{version='2.4.0', protocolVersion='2.4', messageVersion='2.4', zookeeperVersion='3.5.6', isDefault=true, unsupportedFeatures='null'} (no version change)
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:770 - Kafka.spec.kafka.version is unchanged therefore no change to Zookeeper is required
2020-03-19 08:35:36 DEBUG ServiceAccountOperator:102 - ServiceAccount nonprod-seller-kfk/seller-zookeeper already exists, patching it
2020-03-19 08:35:36 DEBUG PvcOperator:102 - PersistentVolumeClaim nonprod-seller-kfk/data-seller-zookeeper-0 already exists, patching it
2020-03-19 08:35:36 DEBUG PvcOperator:102 - PersistentVolumeClaim nonprod-seller-kfk/data-seller-zookeeper-2 already exists, patching it
2020-03-19 08:35:36 DEBUG PvcOperator:102 - PersistentVolumeClaim nonprod-seller-kfk/data-seller-zookeeper-1 already exists, patching it
2020-03-19 08:35:36 DEBUG PvcOperator:168 - PersistentVolumeClaim data-seller-zookeeper-1 in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:36 DEBUG PvcOperator:168 - PersistentVolumeClaim data-seller-zookeeper-0 in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:36 DEBUG PvcOperator:168 - PersistentVolumeClaim data-seller-zookeeper-2 in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:36 DEBUG ServiceOperator:102 - Service nonprod-seller-kfk/seller-zookeeper-client already exists, patching it
2020-03-19 08:35:36 DEBUG ServiceOperator:168 - Service seller-zookeeper-client in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:36 DEBUG ServiceOperator:102 - Service nonprod-seller-kfk/seller-zookeeper-nodes already exists, patching it
2020-03-19 08:35:36 DEBUG ServiceOperator:168 - Service seller-zookeeper-nodes in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:36 DEBUG ConfigMapOperator:102 - ConfigMap nonprod-seller-kfk/seller-zookeeper-config already exists, patching it
2020-03-19 08:35:36 DEBUG ConfigMapOperator:47 - ConfigMap seller-zookeeper-config in namespace nonprod-seller-kfk has not been patched because resources are equal
2020-03-19 08:35:36 DEBUG AbstractModel:471 - Generating certificates
2020-03-19 08:35:36 DEBUG AbstractModel:474 - Cluster communication certificates
2020-03-19 08:35:36 DEBUG Ca:137 - cluster-ca: Reconciling zookeeper certificates
2020-03-19 08:35:36 DEBUG Ca:361 - Certificate for seller-zookeeper-0 already exists
2020-03-19 08:35:36 DEBUG Ca:361 - Certificate for seller-zookeeper-1 already exists
2020-03-19 08:35:36 DEBUG Ca:361 - Certificate for seller-zookeeper-2 already exists
2020-03-19 08:35:36 DEBUG AbstractModel:476 - End generating certificates
2020-03-19 08:35:36 DEBUG SecretOperator:102 - Secret nonprod-seller-kfk/seller-zookeeper-nodes already exists, patching it
2020-03-19 08:35:36 DEBUG SecretOperator:168 - Secret seller-zookeeper-nodes in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:36 DEBUG PodDisruptionBudgetOperator:102 - PodDisruptionBudget nonprod-seller-kfk/seller-zookeeper already exists, patching it
2020-03-19 08:35:36 DEBUG PodDisruptionBudgetOperator:149 - PodDisruptionBudget seller-zookeeper in namespace nonprod-seller-kfk has been deleted
2020-03-19 08:35:36 DEBUG PodDisruptionBudgetOperator:195 - PodDisruptionBudget seller-zookeeper in namespace nonprod-seller-kfk has been created
2020-03-19 08:35:36 DEBUG ZookeeperSetOperator:102 - StatefulSet nonprod-seller-kfk/seller-zookeeper already exists, patching it
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/revisionHistoryLimit"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/metadata/annotations/strimzi.io~1generation"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/failureThreshold"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/periodSeconds"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/successThreshold"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/failureThreshold"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/periodSeconds"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/successThreshold"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/resources"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/terminationMessagePath"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/terminationMessagePolicy"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/livenessProbe/failureThreshold"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/livenessProbe/periodSeconds"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/livenessProbe/successThreshold"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/readinessProbe/failureThreshold"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/readinessProbe/periodSeconds"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/readinessProbe/successThreshold"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/resources"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/terminationMessagePath"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/terminationMessagePolicy"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/dnsPolicy"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/restartPolicy"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/serviceAccount"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/template/spec/volumes/0/configMap/defaultMode"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/volumeClaimTemplates/0/spec/volumeMode"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/spec/volumeClaimTemplates/0/status"}
2020-03-19 08:35:36 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-zookeeper ignoring diff {"op":"remove","path":"/status"}
2020-03-19 08:35:36 DEBUG StatefulSetOperator:297 - Patching StatefulSet nonprod-seller-kfk/seller-zookeeper
2020-03-19 08:35:36 DEBUG ZookeeperSetOperator:168 - StatefulSet seller-zookeeper in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:36 DEBUG ZookeeperSetOperator:79 - Considering rolling update of nonprod-seller-kfk/seller-zookeeper
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:2825 - Rolling update of nonprod-seller-kfk/seller-zookeeper: pod seller-zookeeper-0 has strimzi.io/generation=0; sts has strimzi.io/generation=0
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:2843 - Rolling update of nonprod-seller-kfk/seller-zookeeper: pod seller-zookeeper-0 has strimzi.io/custom-cert-tls-listener-thumbprint=; sts has strimzi.io/custom-cert-tls-listener-thumbprint=
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:2843 - Rolling update of nonprod-seller-kfk/seller-zookeeper: pod seller-zookeeper-0 has strimzi.io/custom-cert-external-listener-thumbprint=; sts has strimzi.io/custom-cert-external-listener-thumbprint=
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:2825 - Rolling update of nonprod-seller-kfk/seller-zookeeper: pod seller-zookeeper-1 has strimzi.io/generation=0; sts has strimzi.io/generation=0
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:2843 - Rolling update of nonprod-seller-kfk/seller-zookeeper: pod seller-zookeeper-1 has strimzi.io/custom-cert-tls-listener-thumbprint=; sts has strimzi.io/custom-cert-tls-listener-thumbprint=
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:2843 - Rolling update of nonprod-seller-kfk/seller-zookeeper: pod seller-zookeeper-1 has strimzi.io/custom-cert-external-listener-thumbprint=; sts has strimzi.io/custom-cert-external-listener-thumbprint=
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:2825 - Rolling update of nonprod-seller-kfk/seller-zookeeper: pod seller-zookeeper-2 has strimzi.io/generation=0; sts has strimzi.io/generation=0
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:2843 - Rolling update of nonprod-seller-kfk/seller-zookeeper: pod seller-zookeeper-2 has strimzi.io/custom-cert-tls-listener-thumbprint=; sts has strimzi.io/custom-cert-tls-listener-thumbprint=
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:2843 - Rolling update of nonprod-seller-kfk/seller-zookeeper: pod seller-zookeeper-2 has strimzi.io/custom-cert-external-listener-thumbprint=; sts has strimzi.io/custom-cert-external-listener-thumbprint=
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:2253 - Reconciliation #35(timer) Kafka(nonprod-seller-kfk/seller): Checking readiness of pod seller-zookeeper-0.
2020-03-19 08:35:36 DEBUG Util:50 - Waiting for Pods resource seller-zookeeper-0 in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:2253 - Reconciliation #35(timer) Kafka(nonprod-seller-kfk/seller): Checking readiness of pod seller-zookeeper-1.
2020-03-19 08:35:36 DEBUG Util:50 - Waiting for Pods resource seller-zookeeper-1 in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:36 DEBUG KafkaAssemblyOperator:2253 - Reconciliation #35(timer) Kafka(nonprod-seller-kfk/seller): Checking readiness of pod seller-zookeeper-2.
2020-03-19 08:35:36 DEBUG Util:50 - Waiting for Pods resource seller-zookeeper-2 in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:36 DEBUG Util:72 - Pods resource seller-zookeeper-0 in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG Util:72 - Pods resource seller-zookeeper-1 in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG Util:72 - Pods resource seller-zookeeper-2 in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Endpoints resource seller-zookeeper-client in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG Util:72 - Endpoints resource seller-zookeeper-client in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Endpoints resource seller-zookeeper-nodes in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG Util:72 - Endpoints resource seller-zookeeper-nodes in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:2341 - Reconciliation #35(timer) Kafka(nonprod-seller-kfk/seller): Considering manual cleaning of Pods for StatefulSet seller-kafka
2020-03-19 08:35:37 DEBUG NetworkPolicyOperator:102 - NetworkPolicy nonprod-seller-kfk/seller-network-policy-kafka already exists, patching it
2020-03-19 08:35:37 DEBUG NetworkPolicyOperator:168 - NetworkPolicy seller-network-policy-kafka in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:835 - STS seller-kafka currently has Kafka version KafkaVersion{version='2.4.0', protocolVersion='2.4', messageVersion='2.4', zookeeperVersion='3.5.6', isDefault=true, unsupportedFeatures='null'}
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:845 - STS seller-kafka is moving from Kafka version KafkaVersion{version='2.4.0', protocolVersion='2.4', messageVersion='2.4', zookeeperVersion='3.5.6', isDefault=true, unsupportedFeatures='null'}
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:855 - STS seller-kafka is moving to Kafka version KafkaVersion{version='2.4.0', protocolVersion='2.4', messageVersion='2.4', zookeeperVersion='3.5.6', isDefault=true, unsupportedFeatures='null'}
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:859 - Kafka version change: Kafka version=KafkaVersion{version='2.4.0', protocolVersion='2.4', messageVersion='2.4', zookeeperVersion='3.5.6', isDefault=true, unsupportedFeatures='null'} (no version change)
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:905 - Kafka.spec.kafka.version unchanged
2020-03-19 08:35:37 DEBUG PvcOperator:102 - PersistentVolumeClaim nonprod-seller-kfk/data-seller-kafka-0 already exists, patching it
2020-03-19 08:35:37 DEBUG PvcOperator:102 - PersistentVolumeClaim nonprod-seller-kfk/data-seller-kafka-1 already exists, patching it
2020-03-19 08:35:37 DEBUG PvcOperator:102 - PersistentVolumeClaim nonprod-seller-kfk/data-seller-kafka-2 already exists, patching it
2020-03-19 08:35:37 DEBUG PvcOperator:168 - PersistentVolumeClaim data-seller-kafka-0 in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG PvcOperator:168 - PersistentVolumeClaim data-seller-kafka-2 in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG PvcOperator:168 - PersistentVolumeClaim data-seller-kafka-1 in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG ServiceAccountOperator:102 - ServiceAccount nonprod-seller-kfk/seller-kafka already exists, patching it
2020-03-19 08:35:37 DEBUG ClusterRoleBindingOperator:101 - ClusterRoleBinding strimzi-nonprod-seller-kfk-seller-kafka-init already exists, patching it
2020-03-19 08:35:37 DEBUG ClusterRoleBindingOperator:177 - ClusterRoleBinding strimzi-nonprod-seller-kfk-seller-kafka-init has been patched
2020-03-19 08:35:37 DEBUG ServiceOperator:102 - Service nonprod-seller-kfk/seller-kafka-bootstrap already exists, patching it
2020-03-19 08:35:37 DEBUG ServiceOperator:168 - Service seller-kafka-bootstrap in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG ServiceOperator:102 - Service nonprod-seller-kfk/seller-kafka-brokers already exists, patching it
2020-03-19 08:35:37 DEBUG ServiceOperator:168 - Service seller-kafka-brokers in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG ServiceOperator:102 - Service nonprod-seller-kfk/seller-kafka-external-bootstrap already exists, patching it
2020-03-19 08:35:37 DEBUG ServiceOperator:168 - Service seller-kafka-external-bootstrap in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG ServiceOperator:102 - Service nonprod-seller-kfk/seller-kafka-2 already exists, patching it
2020-03-19 08:35:37 DEBUG ServiceOperator:102 - Service nonprod-seller-kfk/seller-kafka-0 already exists, patching it
2020-03-19 08:35:37 DEBUG ServiceOperator:102 - Service nonprod-seller-kfk/seller-kafka-1 already exists, patching it
2020-03-19 08:35:37 DEBUG ServiceOperator:168 - Service seller-kafka-2 in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG ServiceOperator:168 - Service seller-kafka-0 in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG ServiceOperator:168 - Service seller-kafka-1 in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Service resource seller-kafka-external-bootstrap in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG Util:72 - Service resource seller-kafka-external-bootstrap in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Service resource seller-kafka-0 in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Service resource seller-kafka-1 in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Service resource seller-kafka-2 in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG Util:72 - Service resource seller-kafka-0 in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG Util:72 - Service resource seller-kafka-1 in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG Util:72 - Service resource seller-kafka-2 in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG AbstractModel:726 - Generating certificates
2020-03-19 08:35:37 DEBUG Ca:192 - cluster-ca: Reconciling kafka broker certificates
2020-03-19 08:35:37 DEBUG Ca:361 - Certificate for seller-kafka-0 already exists
2020-03-19 08:35:37 DEBUG Ca:361 - Certificate for seller-kafka-1 already exists
2020-03-19 08:35:37 DEBUG Ca:361 - Certificate for seller-kafka-2 already exists
2020-03-19 08:35:37 DEBUG AbstractModel:734 - End generating certificates
2020-03-19 08:35:37 DEBUG ConfigMapOperator:102 - ConfigMap nonprod-seller-kfk/seller-kafka-config already exists, patching it
2020-03-19 08:35:37 DEBUG ConfigMapOperator:47 - ConfigMap seller-kafka-config in namespace nonprod-seller-kfk has not been patched because resources are equal
2020-03-19 08:35:37 DEBUG SecretOperator:102 - Secret nonprod-seller-kfk/seller-kafka-brokers already exists, patching it
2020-03-19 08:35:37 DEBUG SecretOperator:168 - Secret seller-kafka-brokers in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG SecretOperator:111 - Secret nonprod-seller-kfk/seller-kafka-jmx does not exist, noop
2020-03-19 08:35:37 DEBUG PodDisruptionBudgetOperator:102 - PodDisruptionBudget nonprod-seller-kfk/seller-kafka already exists, patching it
2020-03-19 08:35:37 DEBUG PodDisruptionBudgetOperator:149 - PodDisruptionBudget seller-kafka in namespace nonprod-seller-kfk has been deleted
2020-03-19 08:35:37 DEBUG PodDisruptionBudgetOperator:195 - PodDisruptionBudget seller-kafka in namespace nonprod-seller-kfk has been created
2020-03-19 08:35:37 DEBUG KafkaSetOperator:102 - StatefulSet nonprod-seller-kfk/seller-kafka already exists, patching it
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/revisionHistoryLimit"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/metadata/annotations/strimzi.io~1generation"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/failureThreshold"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/periodSeconds"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/livenessProbe/successThreshold"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/failureThreshold"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/periodSeconds"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/readinessProbe/successThreshold"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/resources"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/terminationMessagePath"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/0/terminationMessagePolicy"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/livenessProbe/failureThreshold"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/livenessProbe/periodSeconds"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/livenessProbe/successThreshold"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/readinessProbe/failureThreshold"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/readinessProbe/periodSeconds"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/readinessProbe/successThreshold"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/resources"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/terminationMessagePath"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/containers/1/terminationMessagePolicy"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/dnsPolicy"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/initContainers/0/env/0/valueFrom/fieldRef/apiVersion"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/initContainers/0/terminationMessagePath"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/initContainers/0/terminationMessagePolicy"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/restartPolicy"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/serviceAccount"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/template/spec/volumes/4/configMap/defaultMode"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/volumeClaimTemplates/0/spec/volumeMode"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/spec/volumeClaimTemplates/0/status"}
2020-03-19 08:35:37 DEBUG StatefulSetDiff:82 - StatefulSet nonprod-seller-kfk/seller-kafka ignoring diff {"op":"remove","path":"/status"}
2020-03-19 08:35:37 DEBUG StatefulSetOperator:297 - Patching StatefulSet nonprod-seller-kfk/seller-kafka
2020-03-19 08:35:37 DEBUG KafkaSetOperator:168 - StatefulSet seller-kafka in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG KafkaRoller:149 - Initial order for rolling restart [0, 1, 2]
2020-03-19 08:35:37 DEBUG KafkaRoller:193 - Considering restart of pod 0 after delay of 0 MILLISECONDS
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:2825 - Rolling update of nonprod-seller-kfk/seller-kafka: pod seller-kafka-0 has strimzi.io/generation=0; sts has strimzi.io/generation=0
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:2843 - Rolling update of nonprod-seller-kfk/seller-kafka: pod seller-kafka-0 has strimzi.io/custom-cert-tls-listener-thumbprint=; sts has strimzi.io/custom-cert-tls-listener-thumbprint=
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:2843 - Rolling update of nonprod-seller-kfk/seller-kafka: pod seller-kafka-0 has strimzi.io/custom-cert-external-listener-thumbprint=; sts has strimzi.io/custom-cert-external-listener-thumbprint=
2020-03-19 08:35:37 DEBUG KafkaRoller:279 - Pod 0 does not need to be restarted
2020-03-19 08:35:37 DEBUG KafkaRoller:280 - Waiting for non-restarted pod 0 to become ready
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Pods resource seller-kafka-0 in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG Util:72 - Pods resource seller-kafka-0 in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG KafkaRoller:282 - Pod 0 is now ready
2020-03-19 08:35:37 DEBUG KafkaRoller:193 - Considering restart of pod 1 after delay of 0 MILLISECONDS
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:2825 - Rolling update of nonprod-seller-kfk/seller-kafka: pod seller-kafka-1 has strimzi.io/generation=0; sts has strimzi.io/generation=0
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:2843 - Rolling update of nonprod-seller-kfk/seller-kafka: pod seller-kafka-1 has strimzi.io/custom-cert-tls-listener-thumbprint=; sts has strimzi.io/custom-cert-tls-listener-thumbprint=
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:2843 - Rolling update of nonprod-seller-kfk/seller-kafka: pod seller-kafka-1 has strimzi.io/custom-cert-external-listener-thumbprint=; sts has strimzi.io/custom-cert-external-listener-thumbprint=
2020-03-19 08:35:37 DEBUG KafkaRoller:279 - Pod 1 does not need to be restarted
2020-03-19 08:35:37 DEBUG KafkaRoller:280 - Waiting for non-restarted pod 1 to become ready
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Pods resource seller-kafka-1 in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG Util:72 - Pods resource seller-kafka-1 in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG KafkaRoller:282 - Pod 1 is now ready
2020-03-19 08:35:37 DEBUG KafkaRoller:193 - Considering restart of pod 2 after delay of 0 MILLISECONDS
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:2825 - Rolling update of nonprod-seller-kfk/seller-kafka: pod seller-kafka-2 has strimzi.io/generation=0; sts has strimzi.io/generation=0
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:2843 - Rolling update of nonprod-seller-kfk/seller-kafka: pod seller-kafka-2 has strimzi.io/custom-cert-tls-listener-thumbprint=; sts has strimzi.io/custom-cert-tls-listener-thumbprint=
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:2843 - Rolling update of nonprod-seller-kfk/seller-kafka: pod seller-kafka-2 has strimzi.io/custom-cert-external-listener-thumbprint=; sts has strimzi.io/custom-cert-external-listener-thumbprint=
2020-03-19 08:35:37 DEBUG KafkaRoller:279 - Pod 2 does not need to be restarted
2020-03-19 08:35:37 DEBUG KafkaRoller:280 - Waiting for non-restarted pod 2 to become ready
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Pods resource seller-kafka-2 in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG Util:72 - Pods resource seller-kafka-2 in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG KafkaRoller:282 - Pod 2 is now ready
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:2253 - Reconciliation #35(timer) Kafka(nonprod-seller-kfk/seller): Checking readiness of pod seller-kafka-0.
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Pods resource seller-kafka-0 in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:2253 - Reconciliation #35(timer) Kafka(nonprod-seller-kfk/seller): Checking readiness of pod seller-kafka-1.
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Pods resource seller-kafka-1 in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:2253 - Reconciliation #35(timer) Kafka(nonprod-seller-kfk/seller): Checking readiness of pod seller-kafka-2.
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Pods resource seller-kafka-2 in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG Util:72 - Pods resource seller-kafka-0 in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG Util:72 - Pods resource seller-kafka-1 in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG Util:72 - Pods resource seller-kafka-2 in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Endpoints resource seller-kafka-bootstrap in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG Util:72 - Endpoints resource seller-kafka-bootstrap in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Endpoints resource seller-kafka-brokers in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG Util:72 - Endpoints resource seller-kafka-brokers in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG ServiceAccountOperator:111 - ServiceAccount nonprod-seller-kfk/seller-topic-operator does not exist, noop
2020-03-19 08:35:37 DEBUG RoleBindingOperator:111 - RoleBinding nonprod-seller-kfk/strimzi-seller-topic-operator does not exist, noop
2020-03-19 08:35:37 DEBUG ConfigMapOperator:111 - ConfigMap nonprod-seller-kfk/seller-topic-operator-config does not exist, noop
2020-03-19 08:35:37 DEBUG SecretOperator:111 - Secret nonprod-seller-kfk/seller-topic-operator-certs does not exist, noop
2020-03-19 08:35:37 DEBUG DeploymentOperator:111 - Deployment nonprod-seller-kfk/seller-topic-operator does not exist, noop
2020-03-19 08:35:37 DEBUG AbstractModel:374 - logging is not set, using default loggers
2020-03-19 08:35:37 DEBUG AbstractModel:374 - logging is not set, using default loggers
2020-03-19 08:35:37 DEBUG ServiceAccountOperator:102 - ServiceAccount nonprod-seller-kfk/seller-entity-operator already exists, patching it
2020-03-19 08:35:37 DEBUG RoleBindingOperator:102 - RoleBinding nonprod-seller-kfk/strimzi-seller-entity-topic-operator already exists, patching it
2020-03-19 08:35:37 DEBUG RoleBindingOperator:168 - RoleBinding strimzi-seller-entity-topic-operator in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG RoleBindingOperator:102 - RoleBinding nonprod-seller-kfk/strimzi-seller-entity-user-operator already exists, patching it
2020-03-19 08:35:37 DEBUG RoleBindingOperator:168 - RoleBinding strimzi-seller-entity-user-operator in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG ConfigMapOperator:102 - ConfigMap nonprod-seller-kfk/seller-entity-topic-operator-config already exists, patching it
2020-03-19 08:35:37 DEBUG ConfigMapOperator:47 - ConfigMap seller-entity-topic-operator-config in namespace nonprod-seller-kfk has not been patched because resources are equal
2020-03-19 08:35:37 DEBUG ConfigMapOperator:102 - ConfigMap nonprod-seller-kfk/seller-entity-user-operator-config already exists, patching it
2020-03-19 08:35:37 DEBUG ConfigMapOperator:47 - ConfigMap seller-entity-user-operator-config in namespace nonprod-seller-kfk has not been patched because resources are equal
2020-03-19 08:35:37 DEBUG SecretOperator:102 - Secret nonprod-seller-kfk/seller-entity-operator-certs already exists, patching it
2020-03-19 08:35:37 DEBUG SecretOperator:168 - Secret seller-entity-operator-certs in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG DeploymentOperator:102 - Deployment nonprod-seller-kfk/seller-entity-operator already exists, patching it
2020-03-19 08:35:37 DEBUG DeploymentOperator:168 - Deployment seller-entity-operator in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Deployment resource seller-entity-operator in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG Util:72 - Deployment resource seller-entity-operator in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Deployment resource seller-entity-operator in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG Util:72 - Deployment resource seller-entity-operator in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG ServiceAccountOperator:102 - ServiceAccount nonprod-seller-kfk/seller-kafka-exporter already exists, patching it
2020-03-19 08:35:37 DEBUG SecretOperator:102 - Secret nonprod-seller-kfk/seller-kafka-exporter-certs already exists, patching it
2020-03-19 08:35:37 DEBUG SecretOperator:168 - Secret seller-kafka-exporter-certs in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG ServiceOperator:102 - Service nonprod-seller-kfk/seller-kafka-exporter already exists, patching it
2020-03-19 08:35:37 DEBUG ServiceOperator:168 - Service seller-kafka-exporter in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG DeploymentOperator:102 - Deployment nonprod-seller-kfk/seller-kafka-exporter already exists, patching it
2020-03-19 08:35:37 DEBUG DeploymentOperator:168 - Deployment seller-kafka-exporter in namespace nonprod-seller-kfk has been patched
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Deployment resource seller-kafka-exporter in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG Util:72 - Deployment resource seller-kafka-exporter in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG Util:50 - Waiting for Deployment resource seller-kafka-exporter in namespace nonprod-seller-kfk to get ready
2020-03-19 08:35:37 DEBUG Util:72 - Deployment resource seller-kafka-exporter in namespace nonprod-seller-kfk is ready
2020-03-19 08:35:37 DEBUG ServiceAccountOperator:111 - ServiceAccount nonprod-seller-kfk/seller-kafka-jmx-trans does not exist, noop
2020-03-19 08:35:37 DEBUG ConfigMapOperator:111 - ConfigMap nonprod-seller-kfk/seller-jmxtrans-config does not exist, noop
2020-03-19 08:35:37 DEBUG DeploymentOperator:111 - Deployment nonprod-seller-kfk/seller-kafka-jmx-trans does not exist, noop
2020-03-19 08:35:37 DEBUG StatusDiff:37 - Ignoring Status diff {"op":"replace","path":"/conditions/0/lastTransitionTime","value":"2020-03-19T08:35:37+0000"}
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:469 - Reconciliation #35(timer) Kafka(nonprod-seller-kfk/seller): Status did not change
2020-03-19 08:35:37 DEBUG KafkaAssemblyOperator:226 - Status for seller is up to date
2020-03-19 08:35:37 INFO  AbstractOperator:262 - Reconciliation #35(timer) Kafka(nonprod-seller-kfk/seller): reconciled
2020-03-19 08:35:37 DEBUG AbstractOperator:184 - Reconciliation #35(timer) Kafka(nonprod-seller-kfk/seller): Lock lock::nonprod-seller-kfk::Kafka::seller released

[carlobongiovanni]

Actually, in your case the operator is listening to all namespaces ... in that case it should be always the only one in the cluster. Maybe I misunderstood and you upgraded from 0.14 to the 0.17.0-rc2?

yes it's the only operator installed. the updated was done progressively: 0.14->0.15->0.16->0.17.0-rc1->0.17.0-rc2 by downloading the released helm chart from github and doing an helm upgrade kafka-operator . -f values.yaml -n kube-system

[scholzj]

@carlobongiovanni If you do kubectl get kmm2 seller-mm2-cluster -o yaml, wat do you get? Can you share the output?

[carlobongiovanni]

this is more interesting, in the status we read connectCluster with alias seller cannot be found in the list of clusters at spec.clusters

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaMirrorMaker2
metadata:
  creationTimestamp: "2020-03-19T11:20:47Z"
  generation: 1
  name: seller-euwe-mm2-cluster
  namespace: nonprod-seller-kfk
  resourceVersion: "130332503"
  selfLink: /apis/kafka.strimzi.io/v1alpha1/namespaces/nonprod-seller-kfk/kafkamirrormaker2s/seller-euwe-mm2-cluster
  uid: ae969f71-69d3-11ea-9729-4201ac15400b
spec:
  clusters:
  - alias: ecg-nonprod
    bootstrapServers: source-lb:9092
  - alias: euwe-seller-nonprod
    authentication:
      passwordSecret:
        password: password
        secretName: mm2-connect
      type: scram-sha-512
      username: mm2-connect
    bootstrapServers: target-lb:9094
    config:
      config.storage.replication.factor: 2
      offset.storage.replication.factor: 2
      status.storage.replication.factor: 2
    tls:
      trustedCertificates:
      - certificate: ca.crt
        secretName: seller-cluster-ca-cert
  connectCluster: seller
  mirrors:
  - checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 1
    groupsPattern: .*
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 1
    sourceCluster: ecg-nonprod
    sourceConnector:
      config:
        offset-syncs.topic.replication.factor: 1
        replication.factor: 1
        sync.topic.acls.enabled: "false"
    targetCluster: euwe-seller-nonprod
    topicsPattern: ^seller.*|^move-seller.*
  replicas: 2
  version: 2.4.0
status:
  conditions:
  - lastTransitionTime: "2020-03-19T11:20:47.647Z"
    message: connectCluster with alias seller cannot be found in the list of clusters
      at spec.clusters
    reason: InvalidResourceException
    status: "True"
    type: NotReady
  observedGeneration: 1

[scholzj]

Yeah, this is wrong: connectCluster: seller ... I will have a look what should be the right value.

[carlobongiovanni]

ok I now renamed the alias of the target cluster in the mm2 resource to seller and now it's creating the mm2 deployment again

[scholzj]

Ok, looking at some example I used in the past, I think this should be the targetCluster. So you should have something like:

connectCluster: euwe-seller-nonprod

This besially defines the Kafka cluster to which does the underlying Kafka Connect connect as its home clustr and creates the config, offset topics etc.

[scholzj]

IIRC the MM2 is right now source connectors only. So it always has to be the target cluster. But in the future, there might be sink connectors as well. So this is a bit of a preparation.

@PaulRMellor Could you check if the docs are clear enough about what the value of the connectCluster field should be?

[carlobongiovanni]

and I also noticed that the value in mirrors.targetCluster which comes later in the mm2 configuration needs to also have the same value of the target cluster.. this is understandable, I probably misunderstood the docs.

[carlobongiovanni]

Thanks a lot, I now understood something more of how to debug problems with strimzi 👍

Closing because all works as expected.. but maybe would you consider of adding a more meaningful message to the operator in case a resource like mm2 can't be created? Or eventually bring the status of the mm2 resource into the output of the operator?

[scholzj]

You are right ... that is why I didn't saw it right from th ebeginning. Normally we throw an exception when something goes wrong. So it is clearly visible from log. Here we don't seem to do so. It hsould throw InvalidResourceException or something like that.

[sknot-rh]

I will add some reasonable logging to ensure we will see the error in the logs next time. Thanks :)