bitnami / containers

Bitnami container images
https://bitnami.com
Other
3.38k stars 4.85k forks source link

[bitnami/kafka] Kraft Mode upgrading kafka from 3.2 to 3.3 #34402

Closed sowjanya007b closed 1 year ago

sowjanya007b commented 1 year ago

Name and Version

3.3

What architecture are you using?

amd64

What steps will reproduce the bug?

We are using the 3 on prem ubuntu VMs to run a container bitnami/kafka:3.3 on each node. We are using the kraft mode and enabled SSL on broker listener (used for clients and producers as well) and controller SSL, but facing issues with SSL on controller. We are using this configuration,

then we are getting this error in logs

 ERROR Closing socket for 10.xx.xxx.xxx:9093-10.xx.xxx.xxx:1542-104 because of error (kafka.network.Processor)
org.apache.kafka.common.errors.InvalidRequestException: Received request api key METADATA which is not enabled
 ERROR Exception while processing request from 10.xx.xxx.xxx:9093-10.xx.xxx.xxx:1542-104 (kafka.network.Processor)
org.apache.kafka.common.errors.InvalidRequestException: Received request api key METADATA which is not enabled

What is the expected behavior?

No response

What do you see instead?

ERROR Closing socket for 10.xx.xxx.xxx:9093-10.xx.xxx.xxx:1542-104 because of error (kafka.network.Processor) org.apache.kafka.common.errors.InvalidRequestException: Received request api key METADATA which is not enabled ERROR Exception while processing request from 10.xx.xxx.xxx:9093-10.xx.xxx.xxx:1542-104 (kafka.network.Processor) org.apache.kafka.common.errors.InvalidRequestException: Received request api key METADATA which is not enabled

Additional information

No response

javsalgar commented 1 year ago

Hi,

Could you share the configuration of each node?

sowjanya007b commented 1 year ago

hi @javsalgar ,we have 3 VMs with docker installed on them and on each of the nodes we have unit file named kafka.service in /etc/systemd/system/ , and in there we have this above configuration and we do not modify anything inside the container. and we have our keystore and truststore in .jks formats in /var/lib/kafka/config/certs folder as described in the documentation

docker run --net=host \
--ulimit nofile="65536" \
-v /var/lib/kafka:/bitnami/kafka \
-p 9092:9092 \
-p 9093:9093 \
-p 9999:9999 \
-p 8083:8083 \
-p 8888:8888 \
-p 8889:8889 \
-e TAKE_FILE_OWNERSHIP="yes" \
-e KAFKA_CFG_DELETE_TOPIC_ENABLE="true" \
-e KAFKA_CFG_MIN_INSYNC_REPLICAS="2" \
-e KAFKA_CFG_NUM_NETWORK_THREADS="3" \
-e KAFKA_CFG_NUM_IO_THREADS="8" \
-e KAFKA_CFG_SOCKET_SEND_BUFFER_BYTES="102400" \
-e KAFKA_CFG_SOCKET_RECEIVE_BUFFER_BYTES="102400" \
-e KAFKA_CFG_SOCKET_REQUEST_MAX_BYTES="369295618" \
-e KAFKA_CFG_NUM_PARTITIONS="6" \
-e KAFKA_CFG_DEFAULT_REPLICATION_FACTOR="3" \
-e KAFKA_CFG_NUM_RECOVERY_THREADS_PER_DATA_DIR="1" \
-e KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR="3" \
-e KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR="3" \
-e KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR="2" \
-e KAFKA_CFG_LOG_RETENTION_HOURS="168" \
-e KAFKA_CFG_LOG_SEGMENT_BYTES="1073741824" \
-e KAFKA_CFG_LOG_RETENTION_CHECK_INTERVAL_MS="300000" \
-e KAFKA_CFG_GROUP_INITIAL_REBALANCE_DELAY_MS="3000" \
-e ALLOW_PLAINTEXT_LISTENER="yes" \
-e KAFKA_INTER_BROKER_USER="XXX" \
-e KAFKA_INTER_BROKER_PASSWORD="XXX" \
-e KAFKA_CLIENT_USERS="XXX" \
-e KAFKA_CLIENT_PASSWORDS="XXX" \
-e KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE="false" \
-e KAFKA_HEAP_OPTS="-Xms4G -Xmx4G" \
-e KAFKA_CFG_LISTENERS=SSL://:9092,CONTROLLER://:9093 \
-e KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:SSL,SSL:SSL \
-e KAFKA_CFG_SECURITY_INTER_BROKER_PROTOCOL=SSL \
-e KAFKA_CFG_SSL_KEYSTORE_PASSWORD=XXX \
-e KAFKA_CFG_SSL_KEY_PASSWORD=XXX \
-e KAFKA_CFG_SSL_KEYSTORE_TYPE=JKS \
-e KAFKA_CFG_SSL_KEYSTORE_LOCATION=/path/to/keystore \
-e KAFKA_CFG_SSL_TRUSTSTORE_LOCATION=/path/to/truststore \
-e KAFKA_CFG_SSL_TRUSTSTORE_PASSWORD=XXX \
-e KAFKA_CFG_SSL_TRUSTSTORE_TYPE=JKS \
-e KAFKA_CFG_ALLOW_EVERYONE_IF_NO_ACL_FOUND=true \
-e KAFKA_CFG_SUPER_USERS=User:CN=XXX,OU=XXX,O=XXX,L=XXX,ST=XXX,C=XXX \
-e KAFKA_OPTS=-Djavax.net.debug=all \
-e KAFKA_ENABLE_KRAFT=yes \
-e KAFKA_CFG_PROCESS_ROLES=broker,controller \
-e KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER \
-e KAFKA_CFG_ADVERTISED_LISTENERS=SSL://NODE_NAME:9092 \
-e KAFKA_CFG_SECURITY_PROTOCOL=SSL \
-e KAFKA_BROKER_ID="1" \
-e KAFKA_KRAFT_CLUSTER_ID=XXX \
-e KAFKA_CFG_CONTROLLER_QUORUM_VOTERS="1@NODE_NAME_1:9093,2@NODE_NAME_2:9093,3@NODE_NAME_3:9093" \
-e KAFKA_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Djava.rmi.server.hostname=HOST_IP_ADDRESS -Djava.net.preferIPv4Stack=true" \
--name=kafka \
bitnami/kafka:3.3
sowjanya007b commented 1 year ago

Hi Team , Could you please give me any update on this

fevisera commented 1 year ago

Hi @sowjanya007b, sorry for the delay.

Because of the complexity and customization of your specific use case, I have not been able to replicate the exact conditions of your problem.

Did you try to run it without SSL, to discard other connectivity, certificate, or configuration issues?

OneCricketeer commented 1 year ago

I think your error is caused by CONTROLLER:SSL. This should be CONTROLLER:CONTROLLER so that Raft requests do not use standard Kafka API.

github-actions[bot] commented 1 year ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 1 year ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

FightingFox commented 1 year ago

Good day! I have this problem too.

I use a very simple configuration:

version: '3'

services:
  kafka:
    image: 'bitnami/kafka:3.2-debian-11'
    environment:
      - KAFKA_CFG_NODE_ID=0
      - KAFKA_CFG_PROCESS_ROLES=controller,broker
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@localhost:9093
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092,EXTERNAL://localhost:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
    ports:
      - 9092:9092
      - 9093:9093
      - 9094:9094

And I have:

kafka-kafka-1 | [2023-09-13 12:13:29,634] ERROR Exception while processing request from 192.168.176.2:9093-192.168.176.1:63504-27 (kafka.network.Processor)
kafka-kafka-1 | org.apache.kafka.common.errors.InvalidRequestException: Received request api key METADATA which is not enabled
fevisera commented 1 year ago

Hi,

I tried to reproduce the error using @FightingFox docker-compose, but I could not:

$ docker-compose ps
NAME                IMAGE                         COMMAND                  SERVICE             CREATED             STATUS              PORTS
kafka-kafka-1       bitnami/kafka:3.2-debian-11   "/opt/bitnami/script…"   kafka               16 minutes ago      Up 16 minutes       0.0.0.0:9092-9094->9092-9094/tcp
$ docker-compose logs | grep ERROR
$ docker-compose logs | grep "METADATA which is not enabled"

Am I missing something? Is there any external component that might be affecting the container's operation?

github-actions[bot] commented 1 year ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 1 year ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.