Open JoergSiebahn opened 3 years ago
edenhill
commented
2 years ago I'm not an expert in TLS and we're simply using OpenSSL to set it all up. Maybe you can inspect our use of OpenSSL in https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_ssl.c and https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_cert.c to see if we're doing something wrong. Thanks
JoergSiebahn
commented
2 years ago I'm not an expert in TLS …
I'm not an export in C 🤣 and not an expert with OpenSSL as well 😞 . So it's hard for me to find out how you are using OpenSSL.
I tried another way and skipped the part of your implementation and directly used openssl.
Based on the generated keys, I created for reproducing the situation, I played a little bit with openssl commands and came up with this:
$ ./generate-ssl.sh
Cleaning up
Generating a 2048 bit RSA private key
...
$ openssl verify -verbose -CAfile out/pass-bundle.pem out/kafka-ca1-signed.pem
out/kafka-ca1-signed.pem: OK
$ openssl verify -verbose -CAfile out/fail-bundle.pem out/kafka-ca1-signed.pem
out/kafka-ca1-signed.pem: C = DE, L = Hamburg, O = Demo Org, OU = Demo, CN = kafka
error 7 at 0 depth lookup:certificate signature failure
4674969260:error:04FFF06A:rsa routines:CRYPTO_internal:block type is not 01:/System/Volumes/Data/SWE/macOS/BuildRoots/220e8a1b79/Library/Caches/com.apple.xbs/Sources/libressl/libressl-56.60.2/libressl-2.8/crypto/rsa/rsa_pk1.c:103:
4674969260:error:04FFF072:rsa routines:CRYPTO_internal:padding check failed:/System/Volumes/Data/SWE/macOS/BuildRoots/220e8a1b79/Library/Caches/com.apple.xbs/Sources/libressl/libressl-56.60.2/libressl-2.8/crypto/rsa/rsa_eay.c:680:
4674969260:error:0DFFF006:asn1 encoding routines:CRYPTO_internal:EVP lib:/System/Volumes/Data/SWE/macOS/BuildRoots/220e8a1b79/Library/Caches/com.apple.xbs/Sources/libressl/libressl-56.60.2/libressl-2.8/crypto/asn1/a_verify.c:155:So it seems like this behavior is already in openssl verify. In other contexts, I assume that the multiple certificates in the single file are split and verified separately. At least one of our customer uses such a certificate bundle in Java services for Kafka and MongoDB with common libraries and the connections are verified. But I wonder if such an implementation is worth the effort and code complexity in librdkafka when you have a library like OpenSSL already available.
Description
When certificates for the Kafka brokers are signed with a custom CA, it is required that the signing CA is the first with the same CN in a bundle that is configured with
ssl.ca.location. If the CA is updated, you can't add the new one besides the old one. Either the connection will break before you update the signed certificates or after you update the signed certificates. This way a rolling update is not possible.We are using librdkafka as a library through Confluent Kafka Go and were able to reproduce the issue with kafkacat.
How to reproduce
In the error case a log with crypto error similar to this will appear:
A working example to reproduce the error in a Docker Compose setup is available here.
IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/edenhill/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.
Checklist
IMPORTANT: We will close issues where the checklist has not been completed.
Please provide the following information:
security.protocol=SASL_SSL sasl.mechanism=PLAIN sasl.username=XXX sasl.password=YYY ssl.ca.location=/keys/ca-bundle.pemRed Hat Universal Base Image 8 Minimaldebug=..as necessary) from librdkafka