Closed zapho closed 1 year ago
This issue might be caused by the way the JKS stores are created. The environment provides client and CA certificates along with their private keys and they are used to create the JKS stores.
BITNAMI_JKS_STORES_DEST=/bitnami/kafka/config/certs
mkdir -p ${BITNAMI_JKS_STORES_DEST}
STOREPASS=${KAFKA_CERTIFICATE_PASSWORD'}
KEYSTORE_FILE=${BITNAMI_JKS_STORES_DEST}/kafka.keystore.jks
TRUSTSTORE_FILE=${BITNAMI_JKS_STORES_DEST}/kafka.truststore.jks
CERTDIR=/etc/my-company/pki/app
CERTFILE="$CERTDIR/chain.pem"
KEYFILE="$CERTDIR/key.pem"
CLIENT_CERT_COMBINED="/tmp/client-cert-combined.pem"
cat $CERTFILE $KEYFILE > $CLIENT_CERT_COMBINED
CACERTDIR=/etc/my-company/pki/ca
CACERTFILE="$CACERTDIR/ca.crt.pem"
CAKEYFILE="$CACERTDIR/key.pem"
CA_CERT_COMBINED="/tmp/ca-combined.pem"
cat $CACERTFILE $CAKEYFILE > $CA_CERT_COMBINED
echo using PEM cert $CACERTFILE to create Java trustore in $TRUSTSTORE_FILE
keytool -import -alias root_cert_for_kafka -file $CACERTFILE -keystore $TRUSTSTORE_FILE -storetype JKS -storepass $STOREPASS -noprompt
echo using $KEYFILE and $CERTFILE to create Java keystore in $KEYSTORE_FILE
#openssl pkcs12 -export -inkey "$KEYFILE" -in "$CLIENT_CERT_COMBINED" -password "pass:$STOREPASS" -out "$KEYSTORE_FILE"
keytool -import -alias kafka-client-cert -file $CLIENT_CERT_COMBINED -keystore $KEYSTORE_FILE -storetype JKS -storepass $STOREPASS -noprompt
# 'just' adding the client cert to the Java keystore is not enough, a TLS handshake error occurs when trying to connect
# to the Kafka broker
# the following is an attempt to solve this issue
echo using $CAKEYFILE and $CACERTFILE add CA cert int the Java keystore in $KEYSTORE_FILE
keytool -import -alias ca-cert -file $CA_CERT_COMBINED -keystore $KEYSTORE_FILE -storetype JKS -storepass $STOREPASS -noprompt
Another thing I do not understand: when I do not provide the KAFKA_CERTIFICATE_PASSWORD in the environment variables or set a wrong value, the broker starts just fine, no exception. As the keystores are password-protected, I would expect an error at startup time.
Have you tried passing -keyalg RSA to your keytool commands? Depending on the version of the JDK you got the keytool from, the cert may be getting generated with DSA. Since the version of kafka in the latest image is running with jdk 17, i believe an SSL connection with TLSv1.3 is preferred but DSA is no longer supported for that version of TLS.
Hi @nvp152 Thanks for the answer. The keystores are fine, I used them with another Kafka broker images and could have a working SSL setup.
Hi @zapho,
I haven't been able to reproduce your issue using the following docker-compose:
version: '2'
services:
kafka:
image: 'bitnami/kafka:3.4'
hostname: kafka
ports:
- '9092'
environment:
- BITNAMI_DEBUG=yes
# KRaft settings
- KAFKA_CFG_NODE_ID=0
- KAFKA_CFG_PROCESS_ROLES=controller,broker
- KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka:9094
# Listeners settings
- KAFKA_CFG_ADVERTISED_LISTENERS=SECURED://kafka:9093,INTERBROKER://localhost:9092
- KAFKA_CFG_LISTENERS=SECURED://:9093,CONTROLLER://:9094,INTERBROKER://:9092
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,SECURED:SSL,INTERBROKER:PLAINTEXT
- KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
- KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERBROKER
# SSL settings
- KAFKA_CERTIFICATE_PASSWORD=my_pass
- KAFKA_TLS_TYPE=JKS
- KAFKA_TLS_CLIENT_AUTH=required
- KAFKA_CFG_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=
volumes:
- "./certs:/opt/bitnami/kafka/config/certs"
In my case, I generated the certificates using the following script:
#!/bin/bash
mkdir certs
# Root CA
echo "Creating CA certificate and key"
openssl req -new -x509 -keyout certs/ca.key -out certs/ca.crt -days 365 -subj "/CN=Sample CA/OU=US/O=US/ST=US/C=US" -passout pass:my_pass
echo "Creating Truststore"
keytool -keystore certs/kafka.truststore.jks -alias CARoot -import -file certs/ca.crt -storepass my_pass -keypass my_pass -noprompt
# Node cert
echo "Creating node key"
keytool -keystore certs/kafka.keystore.jks -alias kafka-$i -validity 365 -genkey -keyalg RSA -dname "cn=kafka, ou=US, o=US, c=US" -storepass my_pass -keypass my_pass
echo "Creating certificate sign request"
keytool -keystore certs/kafka.keystore.jks -alias kafka-$i -certreq -file certs/tls.srl -storepass my_pass -keypass my_pass
echo "Signing certificate request using self-signed CA"
openssl x509 -req -CA certs/ca.crt -CAkey certs/ca.key \
-in certs/tls.srl -out certs/tls.crt \
-days 365 -CAcreateserial \
-passin pass:my_pass
echo "Adding Ca certificate to the keystore"
keytool -keystore certs/kafka.keystore.jks -alias CARoot -import -file certs/ca.crt -storepass my_pass -keypass my_pass -noprompt
echo "Adding signed certificate"
keytool -keystore certs/kafka.keystore.jks -alias kafka-$i -import -file certs/tls.crt -storepass my_pass -keypass my_pass -noprompt
# Cleanup
rm certs/tls.crt certs/tls.srl
This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.
Thanks, @migruiz4. This was indeed a certificate generation issue.
Thanks, @migruiz4. This was indeed a certificate generation issue.
But I didn't have any problems using kafkajs and kafka-ui
Hi @wedreamer,
Could you please provide more details? I'm sorry but I do not understand your comment and how it is related to this topic.
If you are experiencing any issues related to the bitnami/kafka
image, please create a new issue explaining your case and provide there all the details so we can give you better assistance.
Name and Version
bitnami/kafka:3.4
What architecture are you using?
amd64
What steps will reproduce the bug?
I'm trying to spin a Kafka broker in Kraft mode using TLS mutual auth for client connection. But it ends up with a SSL handshake failure when starting a client.
When starting this container, there is no error in the logs:
Looking into the container, the PKS files are there and can be accessed (root user):
but what looks suspicious is that the logs show no ssl configuration
There might be a configuration issue but there is no trace in the logs to understand where it could occur.
What is the expected behavior?
The kafka client can successfully connect via the TLS connection to the broker.
What do you see instead?
Running a Kafka client using the same keystore and trustore files used for the broker leads to a handshake issue:
Trying to check the TLS socker leads to the same kind of issue:
Matching log from the broker:
Additional information
Full broker logs