[Bug]: Kraft migration with AWS external loadbalancer gets stuck

jeromeguiard commented 2 days ago

Bug Description

While triggering a migration from zookeeper to Kraft. The controller nodepool, newly created, never gets healthy. The strimzi operator keeps reporting timeout on newly created nodes.

This happens with a bootstrap config that registers the node to an AWS load balancer.

ome ready: io.strimzi.operator.common.TimeoutException: Exceeded timeout of 300000ms while waiting for Pods resource xxx-dev-poc-xxx-dev-poc-controller-4 in namespace kafka to be ready
2024-10-14 14:07:26 WARN  KafkaAgentClient:141 - Reconciliation #635(watch) Kafka(middleware-kafka/xxx-dev-poc): Failed to get broker state
java.lang.RuntimeException: Unexpected HTTP status code: 404
    at io.strimzi.operator.cluster.operator.resource.KafkaAgentClient.doGet(KafkaAgentClient.java:114) ~[io.strimzi.cluster-operator-0.43.0.jar:0.43.0]
    at io.strimzi.operator.cluster.operator.resource.KafkaAgentClient.getBrokerState(KafkaAgentClient.java:135) ~[io.strimzi.cluster-operator-0.43.0.jar:0.43.0]
    at io.strimzi.operator.cluster.operator.resource.KafkaRoller.restartIfNecessary(KafkaRoller.java:438) ~[io.strimzi.cluster-operator-0.43.0.jar:0.43.0]
    at io.strimzi.operator.cluster.operator.resource.KafkaRoller.lambda$schedule$7(KafkaRoller.java:368) ~[io.strimzi.cluster-operator-0.43.0.jar:0.43.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
    at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-10-14 14:07:26 WARN  KafkaRoller:444 - Reconciliation #635(watch) Kafka(middleware-kafka/xxx-dev-poc): Pod xxx-dev-poc-xxx-dev-poc-controller-4 is not ready. We will check if KafkaRoller can do anything about it.

It appears that the controller node remains unhealthy because it can't become ready. I think it is due to it's registration to the load balancer that keep failing. Most likely the port 9092 is not listening, and so the readiness gate on the pod with the controller role never becomes ready from the operator side.

One fix i found was, in the kubernetes service handling the bootstrap description, to add in spec selector:

strimzi.io/broker-role: "true"

to ensure that only nodes with broker role will be included in the load balancer.

Once added, and pod restarted, the migration actually starts

Steps to reproduce

have a running kafka cluster with zookeeper, kraft disabled an active nodepool for broker, a nodepool for controller ready and this configuration for listener

listeners:
- configuration:
    bootstrap:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-scheme: internal
        service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-XXX
        service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
    brokers:
    - annotations:
        service.beta.kubernetes.io/aws-load-balancer-scheme: internal
        service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-XXX
        service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
      broker: 0
    - annotations:
        service.beta.kubernetes.io/aws-load-balancer-scheme: internal
        service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-XXX
        service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
      broker: 1
    - annotations:
        service.beta.kubernetes.io/aws-load-balancer-scheme: internal
        service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-XXX
        service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
      broker: 2
  name: plain
  port: 9092
  tls: false
  type: loadbalancer

2 . start a kraft migration

annotations:
strimzi.io/kraft: migration

observer new nodepool being created and never become ready

Expected behavior

Expecting the migration to trigger as described in the doc

Strimzi version

0.43.0

Kubernetes version

EKS 1.30

Installation method

helm chart

Infrastructure

EKS

Configuration files and logs

No response

Additional context

No response

scholzj commented 2 days ago

The controller nodes do not do any load balancer registration. In fact, even brokers don't do anything like that. The error itself does not seem to make much sense as the pod is either not ready or we try to connect to it. But you should probably start by sharing full logs and configurations of all compoenents.

scholzj commented 2 days ago

PS: If you are using something like this https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/deploy/pod_readiness_gate/ to bind the Pod readiness to the loadbalancers, than you should make sure it is not applied to the controller nodes as they do not access client connections through any loadbalancers.

strimzi / strimzi-kafka-operator