[UTO] Unidirectional Topic Operator does seem to cause disruption when upgrading from 0.38 with BTO

[scholzj]

The Unidirectional Topic Operator (enabled by default in 0.39) seems to cause disruption when upgrading from 0.38 (with BTO enabled by default).

It seems to be reproducible with the following steps (4 times out of 4 tries):

1. Deploy Strimzi 0.38 and enable KafkaNodePools
2. Deploy the Kafka cluster with 3 Zoos, 2 node pol with 2 brokers each, CC enabled (I used this YAML -> this is my usual YAML, nothing was modified specifically for this test so not sure if any option there is causing this)
3. Once the Kafka cluster is running, deploy test clients:

   kubectl apply -f https://raw.githubusercontent.com/scholzj/kafka-test-apps/main/deployment-ssl-client-auth.yaml

4. Deploy Kafka Connect (I used this YAML -> again this is my usual YAML, nothing was modified specifically for this test so not sure if any option there is causing this; If you want to reuse it, you have to update the docker image and the push secret and use yours)
5. Once it is all running, upgrade to 0.39 and observe the upgrade

Expected behavior:

• Everything upgrades fine, no custom resource transitions to any errors

Actual behavior:

• Some KafkaTopic resources transition to error until they recover again few minutes later.

• When observing the topics, I see this:

  k get kt -w
  NAME                                                                                               CLUSTER      PARTITIONS   REPLICATION FACTOR   READY
  connect-cluster-configs                                                                            my-cluster   1            3                    True
  connect-cluster-offsets                                                                            my-cluster   25           3                    True
  connect-cluster-status                                                                             my-cluster   5            3                    True
  consumer-offsets---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a                                        my-cluster   50           3                    True
  kafka-test-apps                                                                                    my-cluster   3            3                    True
  strimzi-store-topic---effb8e3e057afce1ecf67c3f5d8e4e3ff177fc55                                     my-cluster   1            3                    True
  strimzi-topic-operator-kstreams-topic-store-changelog---b75e702040b99be8a9263134de3507fc0cc4017b   my-cluster   1            3                    True
  strimzi.cruisecontrol.metrics                                                                      my-cluster   1            3                    True
  strimzi.cruisecontrol.modeltrainingsamples                                                         my-cluster   32           2                    True
  strimzi.cruisecontrol.partitionmetricsamples                                                       my-cluster   32           2                    True
  timer-topic                                                                                        my-cluster   1            3                    True
  
  ^^^ Original topics created by BTO from Strimzi 0.38
  
  connect-cluster-configs                                                                            my-cluster   1            3                    True
  connect-cluster-configs                                                                            my-cluster   1            3                    False
  timer-topic                                                                                        my-cluster   1            3                    True
  strimzi.cruisecontrol.partitionmetricsamples                                                       my-cluster   32           2                    True
  strimzi.cruisecontrol.modeltrainingsamples                                                         my-cluster   32           2                    True
  strimzi.cruisecontrol.metrics                                                                      my-cluster   1            3                    True
  strimzi-topic-operator-kstreams-topic-store-changelog---b75e702040b99be8a9263134de3507fc0cc4017b   my-cluster   1            3                    True
  strimzi-store-topic---effb8e3e057afce1ecf67c3f5d8e4e3ff177fc55                                     my-cluster   1            3                    True
  kafka-test-apps                                                                                    my-cluster   3            3                    True
  consumer-offsets---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a                                        my-cluster   50           3                    True
  connect-cluster-status                                                                             my-cluster   5            3                    True
  connect-cluster-offsets                                                                            my-cluster   25           3                    True
  connect-cluster-status                                                                             my-cluster   5            3                    False
  timer-topic                                                                                        my-cluster   1            3                    False
  consumer-offsets---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a                                        my-cluster   50           3                    False
  strimzi-topic-operator-kstreams-topic-store-changelog---b75e702040b99be8a9263134de3507fc0cc4017b   my-cluster   1            3                    False
  kafka-test-apps                                                                                    my-cluster   3            3                    False
  connect-cluster-offsets                                                                            my-cluster   25           3                    False
  strimzi.cruisecontrol.metrics                                                                      my-cluster   1            3                    False
  strimzi-store-topic---effb8e3e057afce1ecf67c3f5d8e4e3ff177fc55                                     my-cluster   1            3                    False
  strimzi.cruisecontrol.partitionmetricsamples                                                       my-cluster   32           2                    False
  strimzi.cruisecontrol.modeltrainingsamples                                                         my-cluster   32           2                    False
  
  ^^^ Topics updated almost immediately after the UTO from 0.39 starts. It seems to first add the finalizer and than set it to error.
  
  timer-topic                                                                                        my-cluster   1            3                    True
  connect-cluster-configs                                                                            my-cluster   1            3                    True
  connect-cluster-status                                                                             my-cluster   5            3                    True
  consumer-offsets---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a                                        my-cluster   50           3                    True
  strimzi-store-topic---effb8e3e057afce1ecf67c3f5d8e4e3ff177fc55                                     my-cluster   1            3                    True
  strimzi.cruisecontrol.metrics                                                                      my-cluster   1            3                    True
  strimzi.cruisecontrol.modeltrainingsamples                                                         my-cluster   32           2                    True
  connect-cluster-offsets                                                                            my-cluster   25           3                    True
  kafka-test-apps                                                                                    my-cluster   3            3                    True
  strimzi.cruisecontrol.partitionmetricsamples                                                       my-cluster   32           2                    True
  strimzi-topic-operator-kstreams-topic-store-changelog---b75e702040b99be8a9263134de3507fc0cc4017b   my-cluster   1            3                    True
  
  ^^^ Fxed after few (2?) minutes, probably with next periodical reconciliation?

• The watch of the topcis with full YAMLs can be found here: https://gist.github.com/scholzj/06b6745b2b58f311f2b553d6a4e6ccf4#file-watch-topics-txt
• The DEBUG log from UTO is here: https://gist.githubusercontent.com/scholzj/06b6745b2b58f311f2b553d6a4e6ccf4/raw/d7e88e679b57d447dbaa072fb07aa415364894c2/UTO.log
• It is not clear if this has some impact on the topics in Kafka or just on the KafkaTopic resources. In one case, I saw some issues with a topic in the Connect cluster. But other times I reproduced it it seemed fine. SO not sure if that was a coincidence or not.

[scholzj]

This is possibly a blocker for 0.39.0 release! At least unless we confirm that it is only a cosmetic issue in UTO without any negative impact. In that case it would be ugly, but perhaps not blocking.