[bitnami/kafka] The issue of using externalAccess for brokers in zookeeper mode

[937988621]

Name and Version

bitnami/kafka 25.3.4

What architecture are you using?

None

What steps will reproduce the bug?

When I tried using ExternalAccess in zookeeper mode, there was an error during the installation of help

Are you using any custom parameters or values?

controller:
  replicaCount: 0

broker:
   replicaCount: 2

externalAccess:
  enabled: true
  autoDiscovery:
    enabled: false
  broker:
    service:
      type: NodePort
      nodePorts:
        - 30106
        - 30107
      useHostIPs: true

kraft:
  enabled: false

zookeeper:
  enabled: true
  replicaCount: 3

controller.replicaCount=0

broker.replicaCount=2

externalAccess.enabled=true
externalAccess.autoDiscovery.enabled=false
externalAccess.broker.service.type=NodePort
externalAccess.broker.service.nodePorts[0]=30106
externalAccess.broker.service.nodePorts[1]=30107
externalAccess.broker.service.useHostIPs=true

kraft.enabled=false

zookeeper.enabled=true
zookeeper.replicaCount=3

What is the expected behavior?

No response

What do you see instead?

Error: template: kafka/templates/NOTES.txt:333:4: executing "kafka/templates/NOTES.txt" at <include "kafka.validateValues" .>: error calling include: template: kafka/templates/_helpers.tpl:1016:51: executing "kafka.validateValues" at <fail>: error calling fail: 
VALUES VALIDATION:

kafka: externalAccess.controller.service.loadBalancerNames or externalAccess.controller.service.loadBalancerIPs
    By specifying "externalAccess.enabled=true", "externalAccess.autoDiscovery.enabled=false" and
    "externalAccess.controller.service.type=LoadBalancer" at least one of externalAccess.controller.service.loadBalancerNames
    or externalAccess.controller.service.loadBalancerIPs  must be set and the length of those arrays must be equal

Additional information

No response

[javsalgar]

Hi,

The reason for the issue is that you are using a set of incompatible values, as the message states:

kafka: externalAccess.controller.service.loadBalancerNames or externalAccess.controller.service.loadBalancerIPs
    By specifying "externalAccess.enabled=true", "externalAccess.autoDiscovery.enabled=false" and
    "externalAccess.controller.service.type=LoadBalancer" at least one of externalAccess.controller.service.loadBalancerNames
    or externalAccess.controller.service.loadBalancerIPs  must be set and the length of those arrays must be equal

Could you try, for example, enabling autoDiscovery?

[937988621]

你好，

出现此问题的原因是您使用了一组不兼容的值，如消息所述：

kafka: externalAccess.controller.service.loadBalancerNames or externalAccess.controller.service.loadBalancerIPs
    By specifying "externalAccess.enabled=true", "externalAccess.autoDiscovery.enabled=false" and
    "externalAccess.controller.service.type=LoadBalancer" at least one of externalAccess.controller.service.loadBalancerNames
    or externalAccess.controller.service.loadBalancerIPs  must be set and the length of those arrays must be equal

例如，您可以尝试启用自动发现吗？

Thank you very much for your response. When I open 'externalAccess. autoDiscovery. enabled=true', I will be prompted with the following error message

Error: template: kafka/templates/NOTES.txt:333:4: executing "kafka/templates/NOTES.txt" at <include "kafka.validateValues" .>: error calling include: template: kafka/templates/_helpers.tpl:1016:51: executing "kafka.validateValues" at <fail>: error calling fail: 

VALUES VALIDATION:

kafka: rbac.create
    By specifying "externalAccess.enabled=true" and "externalAccess.autoDiscovery.enabled=true"
    an initContainer will be used to auto-detect the external IPs/ports by querying the
    K8s API. Please note this initContainer requires specific RBAC resources. You can create them
    by specifying "--set rbac.create=true".

So I tried to turn on "rbac. create=true" again, and at this point, the pod status of the broker was abnormal

Besides, let me show you all my parameters

broker
  replicaCount: 2

externalAccess:
  enabled: true 
  autoDiscovery:
    enabled: true
  broker:
    service:
      type: NodePort
      nodePorts
        - 30106
        - 30107
      useHostIPs:true

rbac:
  create: true

kraft:
  enabled: false

zookeeper:
  enabled: true
  replicaCount: 3

The pod status I saw:

pod/kafka-liulonga-1698978462222-broker-0                             0/1     CrashLoopBackOff   2 (16s ago)     44s
pod/kafka-liulonga-1698978462222-broker-1                             0/1     CrashLoopBackOff   2 (21s ago)     44s
pod/kafka-liulonga-1698978462222-exporter-598549b567-mwp7b            0/1     CrashLoopBackOff   2 (23s ago)     44s
pod/kafka-liulonga-1698978462222-zookeeper-0                          1/1     Running            0               44s
pod/kafka-liulonga-1698978462222-zookeeper-1                          1/1     Running            0               44s
pod/kafka-liulonga-1698978462222-zookeeper-2                          1/1     Running            0               44s
service/kafka-liulonga-1698978462222                                  NodePort    3000::65ea        <none>        5555:3489/TCP,9092:31000/TCP,9095:57126/TCP   44s
service/kafka-liulonga-1698978462222-0-external                       NodePort    169.169.164.40    <none>        9094:30106/TCP                                44s
service/kafka-liulonga-1698978462222-1-external                       NodePort    169.169.28.218    <none>        9094:30107/TCP                                44s
service/kafka-liulonga-1698978462222-broker-headless                  ClusterIP   None              <none>        9094/TCP,9092/TCP                             44s
service/kafka-liulonga-1698978462222-metrics                          ClusterIP   3000::bac         <none>        9308/TCP                                      44s
service/kafka-liulonga-1698978462222-zookeeper                        ClusterIP   3000::96da        <none>        2181/TCP,2888/TCP,3888/TCP                    44s
service/kafka-liulonga-1698978462222-zookeeper-headless               ClusterIP   None              <none>        2181/TCP,2888/TCP,3888/TCP                    44s
service/kafka-liulonga-1698978462222-zookeeper-metrics                ClusterIP   3000::4326        <none>        9141/TCP                                      44s
deployment.apps/kafka-liulonga-1698978462222-exporter                 0/1     1            0           44s
replicaset.apps/kafka-liulonga-1698978462222-exporter-598549b567                 1         1         0       44s
statefulset.apps/kafka-liulonga-1698978462222-broker      0/2     44s
statefulset.apps/kafka-liulonga-1698978462222-zookeeper   3/3     44s

Log of container kafka:

kubectl logs -f -n component-ns kafka-liulonga-1698978462222-broker-0 -c kafka

kafka 01:47:36.60 
kafka 01:47:36.60 Welcome to the Bitnami kafka container
kafka 01:47:36.60 Subscribe to project updates by watching https://github.com/bitnami/containers
kafka 01:47:36.61 Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka 01:47:36.61 
kafka 01:47:36.61 INFO  ==> ** Starting Kafka setup **
kafka 01:47:36.70 INFO  ==> ** Kafka setup finished! **

kafka 01:47:36.71 INFO  ==> ** Starting Kafka **
[2023-11-08 01:47:37,500] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2023-11-08 01:47:37,827] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2023-11-08 01:47:37,857] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.lang.IllegalArgumentException: Error creating broker listeners from 'CLIENT://kafka-liulonga-1698978462222-broker-0.kafka-liulonga-1698978462222-broker-headless.component-ns.svc.cluster.local:9092,INTERNAL://kafka-liulonga-1698978462222-broker-0.kafka-liulonga-1698978462222-broker-headless.component-ns.svc.cluster.local:9094,EXTERNAL://100.10.15.114:': Unable to parse EXTERNAL://100.10.15.114: to a broker endpoint
        at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:254)
        at kafka.server.KafkaConfig.effectiveAdvertisedListeners(KafkaConfig.scala:2087)
        at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2199)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:2160)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1569)
        at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:1492)
        at kafka.Kafka$.buildServer(Kafka.scala:72)
        at kafka.Kafka$.main(Kafka.scala:91)
        at kafka.Kafka.main(Kafka.scala)
Caused by: org.apache.kafka.common.KafkaException: Unable to parse EXTERNAL://100.10.15.114: to a broker endpoint
        at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:58)
        at kafka.utils.CoreUtils$.$anonfun$listenerListToEndPoints$6(CoreUtils.scala:251)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
        at scala.collection.TraversableLike.map(TraversableLike.scala:286)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
        at scala.collection.AbstractTraversable.map(Traversable.scala:108)
        at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:251)
        ... 8 more

[xuyf1001]

Hi,

The reason for the issue is that you are using a set of incompatible values, as the message states:

kafka: externalAccess.controller.service.loadBalancerNames or externalAccess.controller.service.loadBalancerIPs
    By specifying "externalAccess.enabled=true", "externalAccess.autoDiscovery.enabled=false" and
    "externalAccess.controller.service.type=LoadBalancer" at least one of externalAccess.controller.service.loadBalancerNames
    or externalAccess.controller.service.loadBalancerIPs  must be set and the length of those arrays must be equal

Could you try, for example, enabling autoDiscovery?

we have tried what you said，but still so。Any other suggestions?

[andresbono]

The validation error is about the controller component but you actually do not care about the controllers, right? You are in "Zookeeper mode": controller.replicaCount=0, zookeeper.enabled=true and kraft.enabled=false.

I think the chart should not run even run that validation that is throwing the error in your scenario. For example adding the (gt (int .Values.controller.replicaCount) 0) condition in the helper:

diff --git a/bitnami/kafka/templates/_helpers.tpl b/bitnami/kafka/templates/_helpers.tpl
index c2c3c1b43d..c48f869e8b 100644
--- a/bitnami/kafka/templates/_helpers.tpl
+++ b/bitnami/kafka/templates/_helpers.tpl
@@ -1135,7 +1135,7 @@ kafka: rbac.create
 {{- define "kafka.validateValues.externalAccessAutoDiscoveryIPsOrNames" -}}
 {{- $loadBalancerNameListLength := len .Values.externalAccess.controller.service.loadBalancerNames -}}
 {{- $loadBalancerIPListLength := len .Values.externalAccess.controller.service.loadBalancerIPs -}}
-{{- if and .Values.externalAccess.enabled (or .Values.externalAccess.controller.forceExpose (not .Values.controller.controllerOnly)) (eq .Values.externalAccess.controller.service.type "LoadBalancer") (not .Values.externalAccess.autoDiscovery.enabled) (eq $loadBalancerNameListLength 0) (eq $loadBalancerIPListLength 0) }}
+{{- if and .Values.externalAccess.enabled (gt (int .Values.controller.replicaCount) 0) (or .Values.externalAccess.controller.forceExpose (not .Values.controller.controllerOnly)) (eq .Values.externalAccess.controller.service.type "LoadBalancer") (not .Values.externalAccess.autoDiscovery.enabled) (eq $loadBalancerNameListLength 0) (eq $loadBalancerIPListLength 0) }}
 kafka: externalAccess.controller.service.loadBalancerNames or externalAccess.controller.service.loadBalancerIPs
     By specifying "externalAccess.enabled=true", "externalAccess.autoDiscovery.enabled=false" and
     "externalAccess.controller.service.type=LoadBalancer" at least one of externalAccess.controller.service.loadBalancerNames

---

To workaround your issue in the meantime, @937988621, @xuyf1001, can you try with externalAccess.autoDiscovery.enabled=false and controller.controllerOnly=true and let us know if it everything works as you would expect?

[jacek-jablonski]

Hi @andresbono I have encountered the same issue with zookeeper mode. I can confirm that setting values externalAccess.autoDiscovery.enabled=false and controller.controllerOnly=true fixes the issue.

[andresbono]

Thank you for confirming it! I have just submitted #20941 to fix the issue, without the need of the workaround.