[bitnami/kafka] kafka 3.5.0 cannot start since No security protocol defined for listener CONTROLLER

[3AceShowHand]

Name and Version

bitnami/kafka:3.5.0

What architecture are you using?

amd64

What steps will reproduce the bug?

Containers:
  kafka:
    Container ID:   containerd://49329da13c58bed856f978a3b79997e134102b20ca97c166249cc05a68e7b785
    Image:          xxx/bitnami/kafka:3.5.0
    Image ID:     xxx/bitnami/kafka@sha256:d1dc7d29a9431bac72318fd1db73230cb3017d7683727519189a8332ba5237d1
    Port:           9092/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 16 Aug 2024 17:05:25 +0800
      Finished:     Fri, 16 Aug 2024 17:05:26 +0800
    Ready:          False
    Restart Count:  7
    Limits:
      cpu:     4
      memory:  4Gi
    Requests:
      cpu:      4
      memory:   4Gi
    Readiness:  tcp-socket :9092 delay=5s timeout=1s period=10s #success=1 #failure=3

    Environment:
      KAFKA_HEAP_OPTS:                             -Xmx512M -Xms512M -XX:ParallelGCThreads=1
      KAFKA_OPTS:                                  -Dlogging.level=INFO
      ALLOW_PLAINTEXT_LISTENER:                    yes
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP:    PLAINTEXT:PLAINTEXT
      KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR:  3
      KAFKA_CFG_ZOOKEEPER_CONNECT:                 downstream-zk.7513151-1-727:2181
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR:      3
      ZK_CONNECT:                                  downstream-zk.7513151-1-727:2181
      KAFKA_CFG_INTER_BROKER_LISTENER_NAME:        PLAINTEXT
      KAFKA_ZOOKEEPER_CONNECT:                     downstream-zk.7513151-1-727:2181
      KAFKA_ENABLE_KRAFT:                          false

What is the expected behavior?

The 3 nodes kafka cluster can be running succesfully.

What do you see instead?

kafka 09:05:25.07 Welcome to the Bitnami kafka container
kafka 09:05:25.07 Subscribe to project updates by watching https://github.com/bitnami/containers
kafka 09:05:25.07 Submit issues and feature requests at https://github.com/bitnami/containers/issues
kafka 09:05:25.07
kafka 09:05:25.07 INFO  ==> ** Starting Kafka setup **
kafka 09:05:25.11 WARN  ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.
kafka 09:05:25.12 INFO  ==> Initializing Kafka...
kafka 09:05:25.12 INFO  ==> No injected configuration files found, creating default config files
kafka 09:05:25.23 INFO  ==> ** Kafka setup finished! **

kafka 09:05:25.24 INFO  ==> ** Starting Kafka **
[2024-08-16 09:05:25,793] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2024-08-16 09:05:26,080] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2024-08-16 09:05:26,090] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.lang.IllegalArgumentException: Error creating broker listeners from 'PLAINTEXT://:9092,CONTROLLER://:9093': No security protocol defined for listener CONTROLLER
        at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:254)
        at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:233)
        at kafka.server.KafkaConfig.listeners(KafkaConfig.scala:2053)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1772)
        at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1569)
        at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:1492)
        at kafka.Kafka$.buildServer(Kafka.scala:72)
        at kafka.Kafka$.main(Kafka.scala:91)
        at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.IllegalArgumentException: No security protocol defined for listener CONTROLLER
        at kafka.cluster.EndPoint$.$anonfun$createEndPoint$2(EndPoint.scala:49)
        at scala.collection.immutable.Map$Map1.getOrElse(Map.scala:168)
        at kafka.cluster.EndPoint$.securityProtocol$1(EndPoint.scala:49)
        at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:54)
        at kafka.utils.CoreUtils$.$anonfun$listenerListToEndPoints$6(CoreUtils.scala:251)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
        at scala.collection.TraversableLike.map(TraversableLike.scala:286)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
        at scala.collection.AbstractTraversable.map(Traversable.scala:108)
        at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:251)
        ... 8 more

Additional information

We do not want to enable the kraft, so add the environment variable KAFKA_ENABLE_KRAFT: false

[3AceShowHand]

At the very first, the environment looks like this:

            ALLOW_PLAINTEXT_LISTENER: yes
            KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR: "3"
            KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR: "2"
            KAFKA_CFG_ZOOKEEPER_CONNECT: downstream-zk.${testbed}:2181
            KAFKA_HEAP_OPTS: -Xmx512M -Xms512M -XX:ParallelGCThreads=1
            KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "3"
            KAFKA_OPTS: -Dlogging.level=INFO
            KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: "2"
            KAFKA_ZOOKEEPER_CONNECT: downstream-zk.${testbed}:2181

We meet the error

kafka server: Replication-factor is invalid - Unable to replicate the partition 3 time(s): The target replication factor of 3 cannot be reached because only 1 broker(s) are registered.

By reading the documention, I think it may caused by the kraft mode, so add KAFKA_ENABLE_KRAFT=false, we hope to use the zk mode.

After this change, we meet this error

java.lang.IllegalArgumentException: Error creating broker listeners from 'PLAINTEXT://:9092,CONTROLLER://:9093': No security protocol defined for listener CONTROLLER

By reading the #43918, change the environment variable, still not work.

[javsalgar]

Hi!

Could you share a docker-compose.yml that triggers the issue? So we can easily reproduce

[3AceShowHand]

This is fixed by set environment variable, close this issue:

      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT"

[3AceShowHand]

The minimum reproduce docker compose file looks like this:

version: '3.7'

services:
  zookeeper:
    image: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    volumes:
      - /tmp/zookeeper/data:/data
      - /tmp/zookeeper/datalog:/datalog
      - /tmp/zookeeper/logs:/logs
    restart: always

  kafka1:
    image: registry-mirror.pingcap.net/bitnami/kafka:3.5.0
    depends_on:
      - zookeeper
    container_name: kafka1
    ports:
      - 9092:9092
    environment:
      KAFKA_HEAP_OPTS: "-Xmx512M -Xms512M -XX:ParallelGCThreads=1"
      KAFKA_OPTS: "-Dlogging.level=INFO"
      KAFKA_CFG_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      ALLOW_PLAINTEXT_LISTENER: "yes"
      KAFKA_CFG_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT"
      KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR: "3"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "3"
      KAFKA_ENABLE_KRAFT: "false"
    volumes:
      - /tmp/kafka1/data:/data/kafka-data
    restart: unless-stopped

[3AceShowHand]

This is fixed by set environment variable, close this issue:

      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: "PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT"

I am curious why I have to set this environment variable to make the zookeeper mode works.

[javsalgar]

Hi,

It is not clear to me that this issue is related with the Bitnami packaging of Kafka. It seems more that something directly related with how Kafka is implemented. I would check with the upstream Kafka devs

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.