Topic operator throws InvalidStateStoreException repeatedly, and doesn't recover

LiamClarkeNZ commented 2 years ago

Describe the bug

At times the topic operator, when using the Kafka Streams state store, will hit a state where it repeatedly throws a Streams InvalidStateStoreException with the message: "The state store, topic-store, may have migrated to another instance".

To Reproduce

One method is to delete the changelog topic from underneath the Topic Store. Another involves using min.isr and replica counts of those Streams created internal topics to prevent writing to them.

This latter method was observed by a Strimzi user who was using an old version of Topic Operator which correspondingly used an older version of Kafka Streams where replication factor for internal topics was defaulted by Streams to 1.

When Kafka Streams autocreated the changlog topic, it used the RF of 1, but the topic picked up the broker's default min.ISR of 2, thus the topic could never be written to, which caused Kafka Streams to enter an error state. TO 0.26+ uses a Kafka Streams library that changes this default to "use the cluster defaults".

Expected behavior

If the Kafka Streams instance being used for the state store enters an error state, the topic operator:

Produces log statements useful in debugging the failure
Restarts either the Kafka Stream state store, or itself

Environment (please complete the following information):

Strimzi version: 0.22.0 onwards
Installation method: multiple
Kubernetes cluster: multiple
Infrastructure: multiple

YAML files and logs

2022-04-11 09:43:13,84803 INFO  [OkHttp https://10.115.0.1/...] K8sTopicWatcher:50 - 272814|kube +a-topic|2279692560: event ADDED on resource a-topic generation=1, labels={name=a-topic, strimzi.io/cluster=kafka-cluster, team=infrastructure}
2022-04-11 09:43:13,86780 INFO  [kubernetes-ops-pool-16] CrdOperator:112 - Status of KafkaTopic a-topic in namespace kafka has been updated
2022-04-11 09:43:13,87124 ERROR [vert.x-eventloop-thread-0] K8sTopicWatcher:63 - 272814|kube +a-topic|2279692560: Failure processing KafkaTopic watch event ADDED on resource a-topic with labels {name=a-topic, strimzi.io/cluster=kafka-cluster, team=infrastructure}: The state store, topic-store, may have migrated to another instance.
org.apache.kafka.streams.errors.InvalidStateStoreException: The state store, topic-store, may have migrated to another instance.
    at org.apache.kafka.streams.state.internals.WrappingStoreProvider.stores(WrappingStoreProvider.java:54) ~[org.apache.kafka.kafka-streams-2.6.0.jar:?]
    at org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStore.get(CompositeReadOnlyKeyValueStore.java:52) ~[org.apache.kafka.kafka-streams-2.6.0.jar:?]
    at jdk.internal.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
    at io.strimzi.operator.topic.LocalStoreAndServiceFactory$LazyInvocationHandler.invoke(LocalStoreAndServiceFactory.java:60) ~[io.strimzi.topic-operator-0.22.1.jar:0.22.1]
    at com.sun.proxy.$Proxy23.get(Unknown Source) ~[?:?]
    at io.strimzi.operator.topic.KafkaStreamsTopicStore.read(KafkaStreamsTopicStore.java:74) ~[io.strimzi.topic-operator-0.22.1.jar:0.22.1]
    at io.strimzi.operator.topic.TopicOperator.getFromTopicStore(TopicOperator.java:1438) ~[io.strimzi.topic-operator-0.22.1.jar:0.22.1]
    at io.strimzi.operator.topic.TopicOperator.reconcileOnResourceChange(TopicOperator.java:1073) ~[io.strimzi.topic-operator-0.22.1.jar:0.22.1]
    at io.strimzi.operator.topic.TopicOperator$5.lambda$execute$0(TopicOperator.java:1064) ~[io.strimzi.topic-operator-0.22.1.jar:0.22.1]
    at io.vertx.core.Future.lambda$compose$3(Future.java:368) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.FutureImpl.dispatch(FutureImpl.java:105) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.FutureImpl.tryComplete(FutureImpl.java:150) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.FutureImpl.complete(FutureImpl.java:111) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:176) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:21) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:327) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) ~[io.vertx.vertx-core-3.9.1.jar:3.9.1]
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [io.netty.netty-transport-4.1.60.Final.jar:4.1.60.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.60.Final.jar:4.1.60.Final]
    at java.lang.Thread.run(Thread.java:834) [?:?]
2022-04-11 09:43:13,87235 WARN  [vert.x-eventloop-thread-0] TopicOperator:125 - Failure processing KafkaTopic watch event ADDED on resource a-topic with labels {name=a-topic, strimzi.io/cluster=kafka-cluster, team=infrastructure}: The state store, topic-store, may have migrated to another instance.
2022-04-11 09:43:46,28636 INFO  [vert.x-eventloop-thread-1] TopicOperator:1249 - Starting periodic  reconciliation

speedfl commented 1 year ago

Hello. Is there any solution or workaround ?

LiamClarkeNZ commented 1 year ago

Hello. Is there any solution or workaround ?

Hi @speedfl - generally the issue seems to happen when the Topic Operator is very resource constrained - try increasing the cpu request/limit, see if that resolves it.

speedfl commented 1 year ago

Thanks a lot for your fast answer. I will try it!

voarsh2 commented 1 year ago

Hmmm. I am getting this issue too.

LiamClarkeNZ commented 1 year ago

@voarsh2 - try bumping your resource limits also.

crisblancog commented 1 year ago

Facing a similar issue here and bumping the resources didn't work, in fact monitoring the container with grafana we seen that the container is not even using the 10% of the CPU available:

Environment:

Strimzi version: 0.29.0 Kafka version: 3.2.0 Installation method: yaml files Kubernetes cluster: Openshift 4.9 Infrastructure: IBMCloud

Describe the bug

Every time a topic is created, another autocreated topic gets created and neither of them get Ready

YAML and Logs

~$ oc -n c-kafka-sit get kt | grep test-topic
test-topic                                                                                                                                              c-kafka-cluster   1            3
test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687                                                                                                   c-kafka-cluster   1            3

And if I try to delete them the original one can get deleted but the auto created one is completely stuck:

~$ oc -n c-kafka-sit delete kt test-topic test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687
kafkatopic.kafka.strimzi.io "test-topic" deleted
kafkatopic.kafka.strimzi.io "test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687" deleted

~$ oc -n c-kafka-sit get kt | grep test-topic
test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687                                                                                                   c-kafka-cluster   1            3

Topic .yaml

apiVersion: "kafka.strimzi.io/v1beta2"
kind: "KafkaTopic"
metadata:
  labels:
    strimzi.io/cluster: "c-kafka-cluster"
  name: "test-topic"
spec:
  partitions: 1
  replicas: 3
  config:
    min.insync.replicas: 2
    retention.ms: 4838400000
  topicName: "test_topic"

Topics descriptions:


Events:
  Type     Reason  Age        From                                     Message
  ----     ------  ----       ----                                     -------
  Warning          <unknown>  io.strimzi.operator.topic.TopicOperator  io.strimzi.operator.topic.TopicStore$InvalidStateException
  Warning          <unknown>  io.strimzi.operator.topic.TopicOperator  Failure processing KafkaTopic watch event ADDED on resource test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687 with labels {strimzi.io/cluster=c-kafka-cluster}: null
  Warning          <unknown>  io.strimzi.operator.topic.TopicOperator  io.strimzi.operator.topic.TopicStore$InvalidStateException
  Warning          <unknown>  io.strimzi.operator.topic.TopicOperator  io.strimzi.operator.topic.TopicStore$InvalidStateException
  Warning          <unknown>  io.strimzi.operator.topic.TopicOperator  io.strimzi.operator.topic.TopicStore$InvalidStateException
  Warning          <unknown>  io.strimzi.operator.topic.TopicOperator  io.strimzi.operator.topic.TopicStore$InvalidStateException
  Warning          <unknown>  io.strimzi.operator.topic.TopicOperator  io.strimzi.operator.topic.TopicStore$InvalidStateException
  Warning          <unknown>  io.strimzi.operator.topic.TopicOperator  io.strimzi.operator.topic.TopicStore$InvalidStateException
  Warning          <unknown>  io.strimzi.operator.topic.TopicOperator  io.strimzi.operator.topic.TopicStore$InvalidStateException

NOTE: Autocreated topics only appear when topicName: include underscores. But in any case now I'm not able to create topics.

Kafka Config:

Name:         c-kafka-cluster
Namespace:    c-kafka-sit
Labels:       app=c-kafka
Annotations:  <none>
API Version:  kafka.strimzi.io/v1beta2
Kind:         Kafka
Spec:
  Entity Operator:
    Topic Operator:
      Jvm Options:
        Java System Properties:
          Name:   log4j2.formatMsgNoLookups
          Value:  true
      Resources:
        Limits:
          Cpu:     2
          Memory:  1Gi
        Requests:
          Cpu:            1
          Memory:         500Mi
      Watched Namespace:  c-kafka-sit
    User Operator:
      Jvm Options:
        Java System Properties:
          Name:   log4j2.formatMsgNoLookups
          Value:  true
      Resources:
        Limits:
          Cpu:     500m
          Memory:  500Mi
        Requests:
          Cpu:            500m
          Memory:         500Mi
      Watched Namespace:  c-kafka-sit
  Kafka:
    Authorization:
      Super Users:
        user-xxx
      Type:  simple
    Config:
      auto.create.topics.enable:                 false
      default.replication.factor:                3
      inter.broker.protocol.version:             3.2
      log.message.format.version:                3.2
      min.insync.replicas:                       2
      offsets.topic.replication.factor:          3
      transaction.state.log.min.isr:             2
      transaction.state.log.replication.factor:  3
    Jvm Options:
      - Xms:  4G
      - Xmx:  4G
    Listeners:
      Authentication:
        Type:  scram-sha-512
      Configuration:
        Use Service Dns Domain:  true
      Name:                      plain
      Port:                      9092
      Tls:                       false
      Type:                      internal
      Authentication:
        Type:  scram-sha-512
      Name:    tls
      Port:    9093
      Tls:     true
      Type:    internal
      Authentication:
        Type:  scram-sha-512
      Configuration:
        Bootstrap:
          Load Balancer IP:  <IP>
        Broker Cert Chain And Key:
          Certificate:  tls.crt
          Key:          tls.key
          Secret Name:  c-kafka-custom-tls-cert
        Brokers:
          Advertised Host:   host1.net
          Advertised Port:   9094
          Broker:            0
          Load Balancer IP:  <IP>
          Advertised Host:   host2.net
          Advertised Port:   9094
          Broker:            1
          Load Balancer IP:  <IP>
          Advertised Host:   host3.net
          Advertised Port:   9094
          Broker:            2
          Load Balancer IP:  <IP>
      Name:                  external
      Port:                  9094
      Tls:                   true
      Type:                  loadbalancer
    Liveness Probe:
      Initial Delay Seconds:  300
      Timeout Seconds:        5
    Metrics Config:
      Type:  jmxPrometheusExporter
      Value From:
        Config Map Key Ref:
          Key:   kafka-metrics-config.yml
          Name:  c-kafka-metrics
    Readiness Probe:
      Initial Delay Seconds:  15
      Timeout Seconds:        5
    Replicas:                 3
    Resources:
      Limits:
        Cpu:     3
        Memory:  8Gi
      Requests:
        Cpu:     2
        Memory:  4Gi
    Storage:
      Type:  jbod
      Volumes:
        Class:         ibmc-block-retain-silver
        Delete Claim:  false
        Id:            0
        Size:          1500Gi
        Type:          persistent-claim
    Template:
      External Bootstrap Service:
        Metadata:
          Annotations:
            service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type:  private
      Per Pod Service:
        Metadata:
          Annotations:
            service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type:  private
      Pod:
        Affinity:
          Pod Anti Affinity:
            Preferred During Scheduling Ignored During Execution:
              Pod Affinity Term:
                Label Selector:
                  Match Labels:
                    app.kubernetes.io/name:  kafka
                Namespaces:
                  c-kafka-sit
                Topology Key:  kubernetes.io/hostname
              Weight:          100
        Metadata:
          Annotations:
            backup.velero.io/backup-volumes:  data-0-c-kafka-cluster-kafka-0,data-0-c-kafka-cluster-kafka-1,data-0-c-kafka-cluster-kafka-2
        Security Context:
    Version:  3.2.0
  Kafka Exporter:
    Enable Sarama Logging:  true
    Group Regex:            .*
    Liveness Probe:
      Initial Delay Seconds:  15
      Timeout Seconds:        5
    Logging:                  debug
    Readiness Probe:
      Initial Delay Seconds:  15
      Timeout Seconds:        5
    Resources:
      Limits:
        Cpu:     500m
        Memory:  500Mi
      Requests:
        Cpu:      500m
        Memory:   500Mi
    Topic Regex:  .*
  Zookeeper:
    Jvm Options:
      - Xms:  1G
      - Xmx:  1G
    Metrics Config:
      Type:  jmxPrometheusExporter
      Value From:
        Config Map Key Ref:
          Key:   zookeeper-metrics-config.yml
          Name:  c-kafka-metrics
    Replicas:    3
    Resources:
      Limits:
        Cpu:     1
        Memory:  2Gi
      Requests:
        Cpu:     1
        Memory:  1Gi
    Storage:
      Class:         ibmc-block-retain-silver
      Delete Claim:  false
      Size:          20Gi
      Type:          persistent-claim
    Template:
      Pod:
        Affinity:
          Pod Anti Affinity:
            Preferred During Scheduling Ignored During Execution:
              Pod Affinity Term:
                Label Selector:
                  Match Labels:
                    app.kubernetes.io/name:  zookeeper
                Namespaces:
                  c-kafka-sit
                Topology Key:  kubernetes.io/hostname
              Weight:          100
        Metadata:
          Annotations:
            backup.velero.io/backup-volumes:  data-c-kafka-cluster-zookeeper-0,data-c-kafka-cluster-zookeeper-1,data-c-kafka-cluster-zookeeper-2
        Security Context:

TopicOperator Logs:

2023-03-09 11:32:38,59228 INFO  [OkHttp https://172.21.0.1/...] K8sTopicWatcher:56 - Reconciliation #208675(kube +test-topic) KafkaTopic(c-kafka-sit/test-topic): event ADDED on resource test-topic generation=1, labels={strimzi.io/cluster=c-kafka-cluster}
2023-03-09 11:32:38,70377 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #208681(kube +test-topic) KafkaTopic(c-kafka-sit/test-topic): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:null, privateTopic:null
2023-03-09 11:32:46,17418 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #208697(kube +test-topic) KafkaTopic(c-kafka-sit/test-topic): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:32:46,21535 INFO  [kubernetes-ops-pool-7] CrdOperator:113 - Reconciliation #208700(kube +test-topic) KafkaTopic(c-kafka-sit/test-topic): Status of KafkaTopic test-topic in namespace c-kafka-sit has been updated
2023-03-09 11:32:46,21558 ERROR [vert.x-eventloop-thread-3] K8sTopicWatcher:69 - Reconciliation #208703(kube +test-topic) KafkaTopic(c-kafka-sit/test-topic): Failure processing KafkaTopic watch event ADDED on resource test-topic with labels {strimzi.io/cluster=c-kafka-cluster}: null
2023-03-09 11:32:46,21607 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #208709(kube +test-topic) KafkaTopic(c-kafka-sit/test-topic): Failure processing KafkaTopic watch event ADDED on resource test-topic with labels {strimzi.io/cluster=c-kafka-cluster}: null
2023-03-09 11:32:46,25154 INFO  [OkHttp https://172.21.0.1/...] K8sTopicWatcher:56 - Reconciliation #208716(kube +test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): event ADDED on resource test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687 generation=1, labels={strimzi.io/cluster=c-kafka-cluster}
2023-03-09 11:32:53,71671 INFO  [kubernetes-ops-pool-10] CrdOperator:113 - Reconciliation #208729(/brokers/topics 1:+test_topic) KafkaTopic(c-kafka-sit/test_topic): Status of KafkaTopic test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687 in namespace c-kafka-sit has been updated
2023-03-09 11:32:53,74621 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #208738(kube +test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:33:01,17400 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #214984(kube +test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:33:01,17421 ERROR [pool-3-thread-1] K8sTopicWatcher:69 - Reconciliation #214988(kube +test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Failure processing KafkaTopic watch event ADDED on resource test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687 with labels {strimzi.io/cluster=c-kafka-cluster}: null
2023-03-09 11:33:01,17462 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #214992(kube +test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Failure processing KafkaTopic watch event ADDED on resource test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687 with labels {strimzi.io/cluster=c-kafka-cluster}: null
2023-03-09 11:33:03,43860 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #218036(periodic kube test-topic) KafkaTopic(c-kafka-sit/test-topic): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:33:08,67793 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #218166(periodic kube test-topic) KafkaTopic(c-kafka-sit/test-topic): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:33:08,70374 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #218525(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:33:16,17432 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #218535(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:35:21,29343 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #227820(periodic kube test-topic) KafkaTopic(c-kafka-sit/test-topic): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:35:28,68145 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #228240(periodic kube test-topic) KafkaTopic(c-kafka-sit/test-topic): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:35:28,69362 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #228309(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:35:36,17380 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #228319(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:37:22,73300 INFO  [OkHttp https://172.21.0.1/...] K8sTopicWatcher:56 - Reconciliation #228324(kube -test-topic) KafkaTopic(c-kafka-sit/test-topic): event DELETED on resource test-topic generation=1, labels={strimzi.io/cluster=c-kafka-cluster}
2023-03-09 11:37:22,80545 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #228330(kube -test-topic) KafkaTopic(c-kafka-sit/test-topic): Reconciling topic test_topic, k8sTopic:null, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:37:22,81540 INFO  [OkHttp https://172.21.0.1/...] K8sTopicWatcher:56 - Reconciliation #228333(kube -test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): event DELETED on resource test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687 generation=1, labels={strimzi.io/cluster=c-kafka-cluster}
2023-03-09 11:37:22,82983 INFO  [OkHttp https://172.21.0.1/...] K8sTopicWatcher:56 - Reconciliation #228338(kube +test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): event ADDED on resource test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687 generation=1, labels={strimzi.io/cluster=c-kafka-cluster}
2023-03-09 11:37:28,67413 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #228345(kube -test-topic) KafkaTopic(c-kafka-sit/test-topic): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:37:28,71374 INFO  [kubernetes-ops-pool-2] CrdOperator:113 - Reconciliation #228349(kube -test-topic) KafkaTopic(c-kafka-sit/test-topic): Status of KafkaTopic test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687 in namespace c-kafka-sit has been updated
2023-03-09 11:37:28,71403 ERROR [vert.x-eventloop-thread-3] K8sTopicWatcher:69 - Reconciliation #228351(kube -test-topic) KafkaTopic(c-kafka-sit/test-topic): Failure processing KafkaTopic watch event DELETED on resource test-topic with labels {strimzi.io/cluster=c-kafka-cluster}: null
2023-03-09 11:37:28,71451 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #228358(kube -test-topic) KafkaTopic(c-kafka-sit/test-topic): Failure processing KafkaTopic watch event DELETED on resource test-topic with labels {strimzi.io/cluster=c-kafka-cluster}: null
2023-03-09 11:37:28,72462 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #228360(kube -test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:37:36,17388 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #228369(kube -test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:37:36,17430 ERROR [pool-3-thread-1] K8sTopicWatcher:69 - Reconciliation #228373(kube -test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Failure processing KafkaTopic watch event DELETED on resource test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687 with labels {strimzi.io/cluster=c-kafka-cluster}: null
2023-03-09 11:37:36,17465 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #228379(kube -test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Failure processing KafkaTopic watch event DELETED on resource test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687 with labels {strimzi.io/cluster=c-kafka-cluster}: null
2023-03-09 11:37:36,27057 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #230828(kube +test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:37:43,67422 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #236742(kube +test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:37:43,67460 ERROR [pool-3-thread-1] K8sTopicWatcher:69 - Reconciliation #236746(kube +test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Failure processing KafkaTopic watch event ADDED on resource test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687 with labels {strimzi.io/cluster=c-kafka-cluster}: null
2023-03-09 11:37:43,67508 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #236754(kube +test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Failure processing KafkaTopic watch event ADDED on resource test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687 with labels {strimzi.io/cluster=c-kafka-cluster}: null
2023-03-09 11:37:44,28274 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #237676(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:37:51,17906 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #237806(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:39:01,06130 INFO  [OkHttp https://172.21.0.1/...] K8sTopicWatcher:56 - Reconciliation #238161(kube +test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): event ADDED on resource test-topic-2 generation=1, labels={strimzi.io/cluster=c-kafka-cluster}
2023-03-09 11:39:01,08069 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #238167(kube +test-topic-2) KafkaTopic(c-kafka-sit/test-topi
-2): Reconciling topic test-topic-2, k8sTopic:nonnull, kafkaTopic:null, privateTopic:null
2023-03-09 11:39:01,09370 INFO  [ZkClient-EventThread-20-localhost:2181] ZkTopicsWatcher:142 - Topics created in ZK for watch 2: [test-topic-2]
2023-03-09 11:39:06,17398 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #238183(kube +test-topic-2) KafkaTopic(c-kafka-sit/test-topi
-2): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:39:06,21190 INFO  [kubernetes-ops-pool-9] CrdOperator:113 - Reconciliation #238186(kube +test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): Status of KafkaTopic test-topic-2 in namespace c-kafka-sit has been updated
2023-03-09 11:39:06,21213 ERROR [vert.x-eventloop-thread-3] K8sTopicWatcher:69 - Reconciliation #238188(kube +test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): Failure processing KafkaTopic watch event ADDED on resource test-topic-2 with labels {strimzi.io/cluster=c-kafka-cluster}: null
2023-03-09 11:39:06,21262 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #238194(kube +test-topic-2) KafkaTopic(c-kafka-sit/test-topi
-2): Failure processing KafkaTopic watch event ADDED on resource test-topic-2 with labels {strimzi.io/cluster=c-kafka-cluster}: null
2023-03-09 11:39:07,22764 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #238199(/brokers/topics 2:+test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): Reconciling topic test-topic-2, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:39:13,67410 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #238209(/brokers/topics 2:+test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:39:13,67457 WARN  [pool-3-thread-1] ZkTopicsWatcher:152 - Reconciliation #238213(/brokers/topics 2:+test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): Error responding to creation of topic test-topic-2
2023-03-09 11:39:55,46982 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #247509(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:39:55,47897 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #247514(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): Reconciling topic test-topic-2, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:40:01,17667 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #247614(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:40:01,18110 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #247784(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:42:08,19334 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #257302(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:42:08,20733 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #257307(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): Reconciling topic test-topic-2, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:42:13,67440 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #257317(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:42:13,68541 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #257677(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:44:18,34431 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #267095(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:44:18,35307 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #267100(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): Reconciling topic test-topic-2, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:44:23,68515 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #267270(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:44:23,69110 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #267470(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:46:28,76995 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #276888(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:46:28,78027 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #276893(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): Reconciling topic test-topic-2, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:46:36,17385 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #277273(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:46:36,17465 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #277293(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:48:40,85830 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #286681(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:48:40,86850 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #286686(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): Reconciling topic test-topic-2, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:48:46,17970 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #286906(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:48:46,18673 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #286936(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:50:52,13485 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #296474(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:50:52,14626 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #296479(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): Reconciling topic test-topic-2, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:50:58,67591 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #296539(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:50:58,68882 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #296899(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:53:03,26708 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #306267(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:53:03,27542 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #306272(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): Reconciling topic test-topic-2, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:53:08,69221 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #306712(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:53:08,69266 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #306732(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:55:12,56314 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #316060(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:55:12,57118 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #316065(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): Reconciling topic test-topic-2, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:55:18,67832 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #316225(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:55:18,68002 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #316295(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:57:25,01819 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #325853(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:57:25,05996 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #325858(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): Reconciling topic test-topic-2, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:57:31,17722 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #325968(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:57:31,18428 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #326228(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:59:35,82029 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #335646(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): Reconciling topic test_topic, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:59:35,84674 INFO  [vert.x-eventloop-thread-2] TopicOperator:577 - Reconciliation #335651(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): Reconciling topic test-topic-2, k8sTopic:nonnull, kafkaTopic:nonnull, privateTopic:null
2023-03-09 11:59:41,17701 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #335761(periodic kube test-topic-2) KafkaTopic(c-kafka-sit/test-topic-2): io.strimzi.operator.topic.TopicStore$InvalidStateException
2023-03-09 11:59:41,19048 WARN  [vert.x-eventloop-thread-3] TopicOperator:134 - Reconciliation #336091(periodic kube test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687) KafkaTopic(c-kafka-sit/test-topic---945993b822ab2e73dc74a003ebd1b92d608e8687): io.strimzi.operator.topic.TopicStore$InvalidStateException

22RC commented 12 months ago

For me the solutions was restart(delete) the entity-operator pod which will trigger a fresh reconciliation.

mirwaisx commented 11 months ago

for me the resources did the trick Hello I had also problems with topic creation with status NotReady and adding resources did fix the issue: instead of

entityOperator:
    topicOperator: {}

DO this:

entityOperator:
    topicOperator: 
      resources:
        requests:
          memory: 256Mi
          cpu: "100m"
        limits:
          memory: 512Mi

here was the error

Conditions:
    Last Transition Time:  2023-08-27T22:17:35.460764078Z
    Message:               The state store, topic-store, may have migrated to another instance.
    Reason:                InvalidStateStoreException
    Status:                True
    Type:                  NotReady

evheniyt commented 9 months ago

In my case, restart of entity-operator fix the issue. I think when people change resources for it, it does the same - recreates (restarts) the pod, so I think tunning resources is not relevant in that case

scholzj commented 8 months ago

The Bidirectional Topic Operator (BTO) has been replaced by the new Unidirectional Topic Operator from Strimzi 0.39. There are no plans to fix any outstanding issues in the old BTO and this issue can be closed.

strimzi / strimzi-kafka-operator

Topic operator throws InvalidStateStoreException repeatedly, and doesn't recover #6671