Docker-compose for Kafka: Unable to set JAAS Config

stammaja commented 8 months ago

Name and Version

bitnami/kafka:3.6

What architecture are you using?

amd64

What steps will reproduce the bug?

When starting docker-compose.yaml with any of the options noted below: 1.

  kafka:
    image: docker.io/bitnami/kafka:3.6
    ports:
      - "9092:9092"
    environment:
      # KRaft settings
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka:9093
      # Listeners
      - KAFKA_ADVERTISED_HOST_NAME=localhost
      - KAFKA_CFG_LISTENERS=LISTENER_PLAINTEXT://:9091,LISTENERSASLEXTERNAL://:9092,LISTENER_CONTROLLER://:9093
      - KAFKA_CFG_ADVERTISED_LISTENERS=LISTENER_PLAINTEXT://localhost:9091,LISTENERSASLEXTERNAL://localhost:9092
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=LISTENER_PLAINTEXT:PLAINTEXT,LISTENERSASLEXTERNAL:SASL_PLAINTEXT,LISTENER_CONTROLLER:PLAINTEXT
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=LISTENER_CONTROLLER
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=LISTENER_PLAINTEXT
      - KAFKA_CFG_SASL_ENABLED_MECHANISMS=PLAIN
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_OPTS="-Djava.security.auth.login.config=/opt/bitnami/kafka/config/kafka_jaas.conf"
      - BITNAMI_DEBUG=true
    volumes:
      - ./kafka/kafka_server_jaas.conf:/opt/bitnami/kafka/config/kafka_jaas.conf
    networks:
      - kafka-net

listenersaslexternal.KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
    serviceName="kafka"
    username="admin"
    password="adminsecret123"
    user_local="local_kafka_pw";
};

This version breaks with

Error: Could not find or load main class '-Djava.security.auth.login.config=.opt.bitnami.kafka.config.kafka_jaas.conf' 2024-03-04T09:38:07.605241644Z Caused by: java.lang.ClassNotFoundException: '-Djava.security.auth.login.config=.opt.bitnami.kafka.config.kafka_jaas.conf'

KAFKA_CFG_OPTS instead of KAFKA_OPTS
KafkaServer instead of listenersaslexternal.KafkaServer
- KAFKA_LISTENER_NAME_LISTENERSASLEXTERNAL_PLAIN_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required serviceName="kafka" username="admin" password="adminsecret123" user_local="local_kafka_pw";

instead of using kafka_jaas.config

Options 2,3,4 behave like there was no variable set, cat /opt/bitnami/kafka/config/server.properties gives result `listener.name.listenersaslexternal.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required user_user="bitnami";

What is the expected behavior?

There is a way to set the listener's jaas config without mounting the whole server.properties file

What do you see instead?

In any of the tested scenarios, cat /opt/bitnami/kafka/config/server.properties gives result listener.name.listenersaslexternal.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required user_user="bitnami";

The application gets 'Authentication failed: Invalid username or password'

Additional information

Only option which works is:

    volumes:
      - ./kafka/config/server.properties:/opt/bitnami/kafka/config/server.properties

with listener.name.listenersaslexternal.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required serviceName="kafka" username="admin" password="adminsecret123" user_local="local_kafka_pw";

github-actions[bot] commented 7 months ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

migruiz4 commented 7 months ago

Hi @stammaja,

Let me shed some light here.

Environment variables prefixed KAFKA_CFG_* means they will be set inside kafka.properties file.
Starting versions 3.5.1-debian-11-r4, 3.4.1-debian-11-r50, 3.3.2-debian-11-r176 and 3.2.3-debian-11-r161 we adopted Kafka recommendations to use JAAS configuration inside kafka.properties . A description of all the changes can be found in the README.md
Mounting kafka_jaas.conf is still supported, and KAFKA_OPTS should be automatically set if the file is mounted: https://github.com/bitnami/containers/blob/f7ff2144f293d73b50f80eebb495d24b31da0495/bitnami/kafka/3.6/debian-12/rootfs/opt/bitnami/scripts/kafka/run.sh#L19-L21

Now, considering the above, I have noticed several issues:

Inline Jaas at kafka.properties won't be omitted if kafka_jaas file was provided, therefore making it useless.
Because env variables KAFKA_CFG_* are rendered into kafka.properties at the beginning of the initialization, it is not possible to override settings such has listener.name.<listener>.plain.sasl.jaas.config, because they will be overridden later on the initialization logic.

To fix this issue, I will release an image with the following changes:

kafka_configure_from_environment_variables will be executed at the end of the initialization logic, instead of at the beginning, so users can override automatically configured settings.
Function kafka_configure_server_jaas will be omitted if kafka_jaas.conf was provided.

Therefore, you should be able to address this issue by either setting KAFKA_CFG_LISTENER_NAME_LISTENERSASLEXTERNAL_PLAIN_SASL_JAAS_CONFIG or by mounting your kafka_jaas.config (no need to set KAFKA_OPTS).

migruiz4 commented 7 months ago

I will let you know once a new version of bitnami/kafka is released including those changes.

github-actions[bot] commented 6 months ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 6 months ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

bitnami / containers