[bitnami/kafka] Kraft Mode upgrading kafka from 3.2 to 3.3

[sowjanya007b]

Name and Version

3.3

What architecture are you using?

amd64

What steps will reproduce the bug?

We are using the 3 on prem ubuntu VMs to run a container bitnami/kafka:3.3 on each node. We are using the kraft mode and enabled SSL on broker listener (used for clients and producers as well) and controller SSL, but facing issues with SSL on controller. We are using this configuration,

then we are getting this error in logs

 ERROR Closing socket for 10.xx.xxx.xxx:9093-10.xx.xxx.xxx:1542-104 because of error (kafka.network.Processor)
org.apache.kafka.common.errors.InvalidRequestException: Received request api key METADATA which is not enabled
 ERROR Exception while processing request from 10.xx.xxx.xxx:9093-10.xx.xxx.xxx:1542-104 (kafka.network.Processor)
org.apache.kafka.common.errors.InvalidRequestException: Received request api key METADATA which is not enabled

What is the expected behavior?

No response

What do you see instead?

ERROR Closing socket for 10.xx.xxx.xxx:9093-10.xx.xxx.xxx:1542-104 because of error (kafka.network.Processor) org.apache.kafka.common.errors.InvalidRequestException: Received request api key METADATA which is not enabled ERROR Exception while processing request from 10.xx.xxx.xxx:9093-10.xx.xxx.xxx:1542-104 (kafka.network.Processor) org.apache.kafka.common.errors.InvalidRequestException: Received request api key METADATA which is not enabled

Additional information

No response

[javsalgar]

Hi,

Could you share the configuration of each node?

[sowjanya007b]

hi @javsalgar ,we have 3 VMs with docker installed on them and on each of the nodes we have unit file named kafka.service in /etc/systemd/system/ , and in there we have this above configuration and we do not modify anything inside the container. and we have our keystore and truststore in .jks formats in /var/lib/kafka/config/certs folder as described in the documentation

docker run --net=host \
--ulimit nofile="65536" \
-v /var/lib/kafka:/bitnami/kafka \
-p 9092:9092 \
-p 9093:9093 \
-p 9999:9999 \
-p 8083:8083 \
-p 8888:8888 \
-p 8889:8889 \
-e TAKE_FILE_OWNERSHIP="yes" \
-e KAFKA_CFG_DELETE_TOPIC_ENABLE="true" \
-e KAFKA_CFG_MIN_INSYNC_REPLICAS="2" \
-e KAFKA_CFG_NUM_NETWORK_THREADS="3" \
-e KAFKA_CFG_NUM_IO_THREADS="8" \
-e KAFKA_CFG_SOCKET_SEND_BUFFER_BYTES="102400" \
-e KAFKA_CFG_SOCKET_RECEIVE_BUFFER_BYTES="102400" \
-e KAFKA_CFG_SOCKET_REQUEST_MAX_BYTES="369295618" \
-e KAFKA_CFG_NUM_PARTITIONS="6" \
-e KAFKA_CFG_DEFAULT_REPLICATION_FACTOR="3" \
-e KAFKA_CFG_NUM_RECOVERY_THREADS_PER_DATA_DIR="1" \
-e KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR="3" \
-e KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR="3" \
-e KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR="2" \
-e KAFKA_CFG_LOG_RETENTION_HOURS="168" \
-e KAFKA_CFG_LOG_SEGMENT_BYTES="1073741824" \
-e KAFKA_CFG_LOG_RETENTION_CHECK_INTERVAL_MS="300000" \
-e KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS="3000" \
-e ALLOW_PLAINTEXT_LISTENER="yes" \
-e KAFKA_INTER_BROKER_USER="XXX" \
-e KAFKA_INTER_BROKER_PASSWORD="XXX" \
-e KAFKA_CLIENT_USERS="XXX" \
-e KAFKA_CLIENT_PASSWORDS="XXX" \
-e KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE="false" \
-e KAFKA_HEAP_OPTS="-Xms4G -Xmx4G" \
-e KAFKA_CFG_LISTENERS=SSL://:9092,CONTROLLER://:9093 \
-e KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:SSL,SSL:SSL \
-e KAFKA_CFG_SECURITY_INTER_BROKER_PROTOCOL=SSL \
-e KAFKA_CFG_SSL_KEYSTORE_PASSWORD=XXX \
-e KAFKA_CFG_SSL_KEY_PASSWORD=XXX \
-e KAFKA_CFG_SSL_KEYSTORE_TYPE=JKS \
-e KAFKA_CFG_SSL_KEYSTORE_LOCATION=/path/to/keystore \
-e KAFKA_CFG_SSL_TRUSTSTORE_LOCATION=/path/to/truststore \
-e KAFKA_CFG_SSL_TRUSTSTORE_PASSWORD=XXX \
-e KAFKA_CFG_SSL_TRUSTSTORE_TYPE=JKS \
-e KAFKA_CFG_ALLOW_EVERYONE_IF_NO_ACL_FOUND=true \
-e KAFKA_CFG_SUPER_USERS=User:CN=XXX,OU=XXX,O=XXX,L=XXX,ST=XXX,C=XXX \
-e KAFKA_OPTS=-Djavax.net.debug=all \
-e KAFKA_ENABLE_KRAFT=yes \
-e KAFKA_CFG_PROCESS_ROLES=broker,controller \
-e KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER \
-e KAFKA_CFG_ADVERTISED_LISTENERS=SSL://NODE_NAME:9092 \
-e KAFKA_CFG_SECURITY_PROTOCOL=SSL \
-e KAFKA_BROKER_ID="1" \
-e KAFKA_KRAFT_CLUSTER_ID=XXX \
-e KAFKA_CFG_CONTROLLER_QUORUM_VOTERS="1@NODE_NAME_1:9093,2@NODE_NAME_2:9093,3@NODE_NAME_3:9093" \
-e KAFKA_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Djava.rmi.server.hostname=HOST_IP_ADDRESS -Djava.net.preferIPv4Stack=true" \
--name=kafka \
bitnami/kafka:3.3

[sowjanya007b]

Hi Team , Could you please give me any update on this

[fevisera]

Hi @sowjanya007b, sorry for the delay.

Because of the complexity and customization of your specific use case, I have not been able to replicate the exact conditions of your problem.

Did you try to run it without SSL, to discard other connectivity, certificate, or configuration issues?

[OneCricketeer]

I think your error is caused by CONTROLLER:SSL. This should be CONTROLLER:CONTROLLER so that Raft requests do not use standard Kafka API.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[FightingFox]

Good day! I have this problem too.

I use a very simple configuration:

version: '3'

services:
  kafka:
    image: 'bitnami/kafka:3.2-debian-11'
    environment:
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@localhost:9093
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092,EXTERNAL://localhost:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
    ports:
      - 9092:9092
      - 9093:9093
      - 9094:9094

And I have:

kafka-kafka-1 | [2023-09-13 12:13:29,634] ERROR Exception while processing request from 192.168.176.2:9093-192.168.176.1:63504-27 (kafka.network.Processor)
kafka-kafka-1 | org.apache.kafka.common.errors.InvalidRequestException: Received request api key METADATA which is not enabled

[fevisera]

Hi,

I tried to reproduce the error using @FightingFox docker-compose, but I could not:

$ docker-compose ps
NAME                IMAGE                         COMMAND                  SERVICE             CREATED             STATUS              PORTS
kafka-kafka-1       bitnami/kafka:3.2-debian-11   "/opt/bitnami/script…"   kafka               16 minutes ago      Up 16 minutes       0.0.0.0:9092-9094->9092-9094/tcp
$ docker-compose logs | grep ERROR
$ docker-compose logs | grep "METADATA which is not enabled"

Am I missing something? Is there any external component that might be affecting the container's operation?

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.