Open PawelKalamba opened 1 year ago
Hi @PawelKalamba,
Do you know the logic and order for starting Kafka brokers by the operator - does it starts one by one and wait for a kind of health and then what's this healthcheck or does it start all at once?
I did some quick checks in the source code but couldn't find an appropriate reasoning about the behavior that you see
Dear @PawelKalamba, what is the version of the Koperator that generates the log LogsFromKafkaOperator.txt I can see cruise-control:2.5.68 which is pretty old one. Can you try our latest release ? https://github.com/banzaicloud/koperator/releases/tag/v0.22.0
Hey @PawelKalamba 👋 , any updates on this would be appreciated. BTW, it would also be great to join our Slack channel where we can better communicate on issues / questions like this so other members in the community are also aware of the potential issues
Describe the bug After deleting unused Envoy connectors broke and restarted brokers stopped replicating to the expected number.
Steps to reproduce the issue:
Expected behavior The Kafka operator should recreate all brokers, and elect a leader and the messages should flow normally.
Additional context We have a cluster set up from 5 brokers (here is the configuration CustomKafkaCluster.txt But only 2 out of 5 were started by an operator This is describe from kubernetes DescribeKafka0.txt Now we have only kafka-0 and kafka-1 and 5 services with only two having updated ports for envoy. We tried to deploy the last working version but with no luck. Same with scaling down kafka-operator zookeper-operator then deleting zookeeper and broker pods and then scaling it up. We also noticed that our KafkaCluster is on neverending ClusterRollingUpgrading
Also, I'm attaching some logs from related pods. LogsFromKafkaOperator.txt LogsFromZKOperator.txt LogsFromInternalEnvoy.txt
We also noticed that some of our partitions don't have elected leader
Topic: TopicName TopicId: topicID PartitionCount: 12 ReplicationFactor: 2 Configs: cleanup.policy=compact,delete,retention.ms=604800000,unclean.leader.election.enable=true Topic: TopicName Partition: 0 Leader: 0 Replicas: 3,0 Isr: 0 Topic: TopicName Partition: 1 Leader: 0 Replicas: 0,4 Isr: 0 Topic: TopicName Partition: 2 Leader: 1 Replicas: 4,1 Isr: 1 Topic: TopicName Partition: 3 Leader: 1 Replicas: 1,2 Isr: 1 Topic: TopicName Partition: 4 Leader: none Replicas: 2,3 Isr: 3 Topic TopicName Partition: 5 Leader: none Replicas: 3,4 Isr: 4 Topic: TopicName Partition: 6 Leader: 0 Replicas: 0,1 Isr: 1,0 Topic: TopicName Partition: 7 Leader: none Replicas: 4,2 Isr: 4 Topic: TopicName Partition: 8 Leader: 1 Replicas: 1,3 Isr: 1 Topic: TopicName Partition: 9 Leader: 0 Replicas: 2,0 Isr: 0 Topic: TopicName Partition: 10 Leader: 1 Replicas: 3,1 Isr: 1 Topic: TopicName Partition: 11 Leader: 0 Replicas: 0,2 Isr: 0
Do you know the logic and order for starting Kafka brokers by the operator - does it starts one by one and wait for a kind of health and then what's this healthcheck or does it start all at once?