Open 0ni0nrings opened 1 year ago
Had to expose 9093 on the kafka-svc after noticing it was trying to connect to kafka-1.kafka-svc.kafka-kraft.svc.cluster.local:9093
, which wasn't mapped. My guess is doughgle/kafka-kraft
changed kafka versions so the controller port changed from kafka 2.8 -> 3.1
Hi @0ni0nrings do you have any update on this issue?
@0ni0nrings @Saveriu
If you create image with the help of given docker file and try to deploy with single replica instance it will work fine without any error but when you increase the instance of kafka broker then it gives error in pods other then kafka-0
$ kubectl logs -f kafka-1 -n kafka-kraft
[2023-02-08 18:09:35,830] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2023-02-08 18:09:36,156] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2023-02-08 18:09:36,226] WARN No meta.properties file under dir /mnt/kafka/1/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2023-02-08 18:09:36,229] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
kafka.common.KafkaException: No meta.properties
found in /mnt/kafka/1 (have you run kafka-storage.sh
to format the directory?)
at kafka.server.BrokerMetadataCheckpoint$.$anonfun$getBrokerMetadataAndOfflineDirs$2(BrokerMetadataCheckpoint.scala:164)
at scala.collection.immutable.List.foreach(List.scala:333)
at kafka.server.BrokerMetadataCheckpoint$.getBrokerMetadataAndOfflineDirs(BrokerMetadataCheckpoint.scala:153)
at kafka.server.KafkaRaftServer$.initializeLogDirs(KafkaRaftServer.scala:151)
at kafka.server.KafkaRaftServer.
and you will get java.net.UnknownHostException: kafka-2.kafka-svc.kafka-kraft.svc.cluster.local error in kafka-0 pod because outher pods are down and kafka-0 is trying to connect with them.
i have made some changes in entrypoint.sh file and created new image and it worked for me to run multiple instance.
Hi, I have the same problem deploying Kafka with kraft in kube cluster using bitnami helm distro (version 23.0.5 ). Setting 1 replica works fine, but once I set more than 1 replica it keeps logging the OP error. Is there any news or workaround for more then 1 replica?
Hi @zMcKracken i have made some changes in entrypoint.sh file and created new image and it worked for me to run multiple instance.
PRAYAG-97 commented
@PRAYAG-97 can you give us the changes you made ? thanks in advance
@Firas-Zarai I have made changes in entrypoint.sh file And Please let me know if there is other solution. Docker file: FROM openjdk:11
ENV KAFKA_VERSION=3.4.1 ENV SCALA_VERSION=2.13 ENV KAFKA_HOME=/opt/kafka ENV PATH=${PATH}:${KAFKA_HOME}/bin
LABEL name="kafka" version=${KAFKA_VERSION}
RUN wget -O /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz https://downloads.apache.org/kafka/${KAFKA_VERSION}/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz \ && tar xfz /tmp/kafka_${SCALA_VERSION}-${KAFKAVERSION}.tgz -C /opt \ && rm /tmp/kafka${SCALA_VERSION}-${KAFKAVERSION}.tgz \ && ln -s /opt/kafka${SCALA_VERSION}-${KAFKA_VERSION} ${KAFKAHOME} \ && rm -rf /tmp/kafka${SCALA_VERSION}-${KAFKA_VERSION}.tgz
COPY ./entrypoint.sh / RUN ["chmod", "+x", "/entrypoint.sh"] ENTRYPOINT ["/entrypoint.sh"]
Entrypoint.sh
NODE_ID=${HOSTNAME:6} LISTENERS="PLAINTEXT://:9092,CONTROLLER://:9093" ADVERTISED_LISTENERS="PLAINTEXT://kafka-$NODE_ID.$SERVICE.$NAMESPACE.svc.cluster.local:9092"
CONTROLLER_QUORUM_VOTERS="" for i in $( seq 0 $REPLICAS); do if [[ $i != $REPLICAS ]]; then CONTROLLER_QUORUM_VOTERS="$CONTROLLER_QUORUM_VOTERS$i@kafka-$i.$SERVICE.$NAMESPACE.svc.cluster.local:9093," else CONTROLLER_QUORUM_VOTERS=${CONTROLLER_QUORUM_VOTERS::-1} fi done
mkdir -p $SHARE_DIR/$NODE_ID
CLUSTER_ID="q9D3LywhQWyj-d6AqBASmA"
if [[ ! -f "$SHARE_DIR/cluster_id" && "$NODE_ID" = "0" ]]; then
echo $CLUSTER_ID > $SHARE_DIR/cluster_id
else
echo $CLUSTER_ID > $SHARE_DIR/cluster_id
fi
sed -e "s+^node.id=.+node.id=$NODE_ID+" \ -e "s+^controller.quorum.voters=.+controller.quorum.voters=$CONTROLLER_QUORUM_VOTERS+" \ -e "s+^listeners=.+listeners=$LISTENERS+" \ -e "s+^advertised.listeners=.+advertised.listeners=$ADVERTISED_LISTENERS+" \ -e "s+^log.dirs=.*+log.dirs=$SHARE_DIR/$NODE_ID+" \ /opt/kafka/config/kraft/server.properties > server.properties.updated \ && mv server.properties.updated /opt/kafka/config/kraft/server.properties
kafka-storage.sh format -t $CLUSTER_ID -c /opt/kafka/config/kraft/server.properties
exec kafka-server-start.sh /opt/kafka/config/kraft/server.properties
Description
When deploying kakfa brokers using the guide as-is, the pods fails to stay running and terminate with
CrashLoopBackOff
✋ I have searched the open/closed issues and my issue is not listed.
Versions
Reproduction Code [Required]
https://github.com/IBM/kraft-mode-kafka-on-kubernetes
Expected behavior
Once the the resources are deployed, the pods should continue to stay in
running
state and kafka cluster should be available for consuming and producing events.Actual behavior
Additional context