[bitnami/kafka] External kafka broker connectivity with ClusterIP opening a different port for each Kafka broker

[venkatne]

I am also getting same issue

kafkacat -b test-kafka.ne-dev-internal.net:9095 -L
Metadata for all topics (from broker 0: test-kafka.ne-dev-internal.net:9095/0):
 3 brokers:
  broker 0 at test-kafka.ne-dev-internal.net:9095 (controller)
  broker 1 at test-kafka.ne-dev-internal.net:9096
  broker 2 at test-kafka.ne-dev-internal.net:9097

But svc opening only 9095

kubectl get svc -n kafka
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
kafka                         ClusterIP   10.100.126.126   <none>        9092/TCP,9095/TCP            113m
kafka-connect                 ClusterIP   10.100.126.220   <none>        8083/TCP                     113m
kafka-controller-0-external   ClusterIP   10.100.215.160   <none>        9095/TCP                     113m
kafka-controller-1-external   ClusterIP   10.100.5.252     <none>        9095/TCP                     74m
kafka-controller-2-external   ClusterIP   10.100.130.54    <none>        9095/TCP                     74m
kafka-controller-headless     ClusterIP   None             <none>        9094/TCP,9092/TCP,9093/TCP   113m

Because of this we are getting error when try to produce messages

root@ip-162-30-10-101:/k8sconfig/k8s-venkat/kafka# kafkacat -b test-kafka.ne-dev-internal.net:9095 -t test
% Auto-selecting Consumer mode (use -P or -C to override)
% ERROR: Local: Broker transport failure: test-kafka.ne-dev-internal.net:9097/2: Connect to ipv4#162.30.22.46:9097 failed: Connection refused (after 0ms in state CONNECT)

Originally posted by @venkatne in https://github.com/bitnami/charts/issues/19429#issuecomment-2050095656

[javsalgar]

Hi!

Could you share some sample values that reproduce the issue?

[venkatne]

Here is the snippet block

listeners:
  client:
    containerPort: 9092
    protocol: PLAINTEXT
  controller:
    name: CONTROLLER
    containerPort: 9093
    protocol: PLAINTEXT
  interbroker:
    containerPort: 9094
    protocol: PLAINTEXT
  external:
    containerPort: 9095
    protocol: PLAINTEXT
sasl:
  enabledMechanisms: PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
  interBrokerMechanism: PLAIN
  controllerMechanism: PLAIN
controller.persistence.size: 200Gi
controller.logPersistence.size: 200Gi
broker.persistence.size: 200Gi
broker.logPersistence.size: 200Gi
externalAccess:
  enabled: true
  autoDiscovery:
    enabled: false
  controller:
    service:
      type: ClusterIP
      domain: test-kafka.ne-dev-internal.net
      ports:
        external: 9095
  broker:
    service:
      type: ClusterIP
      domain: test-kafka.ne-dev-internal.net
      ports:
        external: 9095

[anuragwork1]

I got the same issue in my case as well. Here is the issue I have opened.

https://github.com/bitnami/charts/issues/25192

[jotamartos]

Hi all,

As I just mentioned in the other issue, I've been checking the configuration and saw that this is generated by the EXTERNAL_ACCESS_PORT_AUTOINCREMENT env var

https://github.com/bitnami/charts/blob/main/bitnami/kafka/templates/_helpers.tpl#L809

and the configure_external_access method in the configmap

https://github.com/bitnami/charts/blob/main/bitnami/kafka/templates/scripts-configmap.yaml#L150

Would you like to contribute? You can follow our contributing guidelines and the whole community will benefit from this change. Our team will be more than happy to review the changes.

In the meantime, I'm going to create a task on our side to evaluate the changes but I can't provide you with an ETA on when they will be ready.