Open tittoto opened 3 years ago
i am also facing the same issue
var capem = System.IO.File.ReadAllText("RootCA-SHA256.pem");
var certificatepem = System.IO.File.ReadAllText("SslCertificate.pem");
var keypem = System.IO.File.ReadAllText("SslPrivate.pem");
var consumerConfig = new ConsumerConfig()
{
BootstrapServers = _kafkaConfiguration.BootstrapServers,
SecurityProtocol = SecurityProtocol.Ssl,
GroupId = KafkaConstants.Orchestrator_CG,
SslCaLocation = _kafkaConfiguration.SslCaLocation,
//SslCaPem = capem,
//SslCertificateLocation = _kafkaConfiguration.SslCertificateLocation,
SslCertificatePem = certificatepem,
//SslKeyLocation = _kafkaConfiguration.SslKeyLocation,
SslKeyPem = keypem,
//SslKeyPassword = _kafkaConfiguration.SslKeyPassword,
AutoOffsetReset = AutoOffsetReset.Earliest,
EnableAutoCommit = true,
Debug = _kafkaConfiguration.Debug
};
when i tried with setting Location it is working ,but when tried setting with Pem i am getting below Error SSL handshake failed: ssl/statem/statem_clnt.c:395: error:141A10F4:SSL routines:ossl_statem_client_read_transition:unexpected message: client SSL authentication might be required (see ssl.key.location and ssl.certificate.location and consult the broker logs for more information) (after 665ms in state SSL_HANDSHAKE, 28 identical error(s) suppressed)
Interesting. Could you reproduce this with Debug: "ssl,security"
and provide the logs?
Hi @edenhill Thanks for the response please find the logs i am using confluent-kafka version 1.8.2
%7|1645716194.256|SSL|rdkafka#consumer-1| [thrd:app]: Loading CA certificate from string %7|1645716194.256|SSL|rdkafka#consumer-1| [thrd:app]: Loading public key from file /Users/c0r0765/WorkSpace/API/DevKafka/test_certs/SslCertificate.pem %7|1645716194.261|SSL|rdkafka#consumer-1| [thrd:app]: Loading private key from string %7|1645716194.261|INIT|rdkafka#consumer-1| [thrd:app]: librdkafka v1.8.2 (0x10802ff) rdkafka#consumer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, STRIP STATIC_LINKING CC GXX PKGCONFIG OSXLD LIBDL PLUGINS STATIC_LIB_zlib ZLIB STATIC_LIB_libcrypto STATIC_LIB_libssl SSL SASL_CYRUS STATIC_LIB_libzstd ZSTD HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x200) %3|1645716194.989|FAIL|rdkafka#consumer-1| [thrd:ssl://kafka-355344663-2-1426816988.wus.kafka-v2-weiot-edge-stg.]: ssl://kafka-355344663-2-1426816988.wus.kafka-v2-weiot-edge-stg.ms-df-messaging.stg-az-westus-1.prod.us.walmart.net:9093/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1916: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (brew install openssl) (after 227ms in state SSL_HANDSHAKE) %3|1645716194.989|ERROR|rdkafka#consumer-1| [thrd:ssl://kafka-355344663-2-1426816988.wus.kafka-v2-weiot-edge-stg.]: 1/1 brokers are down %3|1645716194.989|ERROR|rdkafka#consumer-1| [thrd:app]: rdkafka#consumer-1: ssl://kafka-355344663-2-1426816988.wus.kafka-v2-weiot-edge-stg.ms-df-messaging.stg-az-westus-1.prod.us.walmart.net:9093/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1916: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (brew install openssl) (after 227ms in state SSL_HANDSHAKE) %3|1645716195.733|FAIL|rdkafka#consumer-1| [thrd:ssl://kafka-355344663-2-1426816988.wus.kafka-v2-weiot-edge-stg.]: ssl://kafka-355344663-2-1426816988.wus.kafka-v2-weiot-edge-stg.ms-df-messaging.stg-az-westus-1.prod.us.walmart.net:9093/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1916: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (brew install openssl) (after 233ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed) %3|1645716195.733|ERROR|rdkafka#consumer-1| [thrd:app]: rdkafka#consumer-1: ssl://kafka-355344663-2-1426816988.wus.kafka-v2-weiot-edge-stg.ms-df-messaging.stg-az-westus-1.prod.us.walmart.net:9093/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1916: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (brew install openssl) (after 233ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)
It says the public key is still loaded from file, is that intended?
%7|1645716194.256|SSL|rdkafka#consumer-1| [thrd:app]: Loading public key from file /Users/c0r0765/WorkSpace/API/DevKafka/test_certs/SslCertificate.pem
i am getting same error ,i am trying to see any combination works
%7|1645718518.406|SSL|rdkafka#consumer-1| [thrd:app]: Loading CA certificate from string %7|1645718518.406|SSL|rdkafka#consumer-1| [thrd:app]: Loading public key from string %7|1645718518.406|SSL|rdkafka#consumer-1| [thrd:app]: Loading private key from string %7|1645718518.407|INIT|rdkafka#consumer-1| [thrd:app]: librdkafka v1.8.2 (0x10802ff) rdkafka#consumer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, STRIP STATIC_LINKING CC GXX PKGCONFIG OSXLD LIBDL PLUGINS STATIC_LIB_zlib ZLIB STATIC_LIB_libcrypto STATIC_LIB_libssl SSL SASL_CYRUS STATIC_LIB_libzstd ZSTD HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x200) %3|1645718519.154|FAIL|rdkafka#consumer-1| [thrd:ssl://kafka-355344663-2-1426816988.wus.kafka-v2-weiot-edge-stg.]: ssl://kafka-355344663-2-1426816988.wus.kafka-v2-weiot-edge-stg.ms-df-messaging.stg-az-westus-1.prod.us.walmart.net:9093/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1916: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (brew install openssl) (after 225ms in state SSL_HANDSHAKE) %3|1645718519.154|ERROR|rdkafka#consumer-1| [thrd:ssl://kafka-355344663-2-1426816988.wus.kafka-v2-weiot-edge-stg.]: 1/1 brokers are down %3|1645718519.154|ERROR|rdkafka#consumer-1| [thrd:app]: rdkafka#consumer-1: ssl://kafka-355344663-2-1426816988.wus.kafka-v2-weiot-edge-stg.ms-df-messaging.stg-az-westus-1.prod.us.walmart.net:9093/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1916: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (brew install openssl) (after 225ms in state SSL_HANDSHAKE) %3|1645718519.890|FAIL|rdkafka#consumer-1| [thrd:ssl://kafka-355344663-2-1426816988.wus.kafka-v2-weiot-edge-stg.]: ssl://kafka-355344663-2-1426816988.wus.kafka-v2-weiot-edge-stg.ms-df-messaging.stg-az-westus-1.prod.us.walmart.net:9093/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1916: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (brew install openssl) (after 269ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed) %3|1645718519.891|ERROR|rdkafka#consumer-1| [thrd:app]: rdkafka#consumer-1: ssl://kafka-355344663-2-1426816988.wus.kafka-v2-weiot-edge-stg.ms-df-messaging.stg-az-westus-1.prod.us.walmart.net:9093/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1916: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (brew install openssl) (after 269ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)
Maybe it fails to load the full CA chain from the PEM string, but just the first certificate
@edenhill can you help me with what can i do to get this working
@lakshmisivareddy Is this still an issue. Can you try passing the location of your CA in the SslCaLocation
in the config?
I see from above comments that you are passing it, checking further.
@lakshmisivareddy @tittoto Can any of you confirm if this is still an issue? With the latest release 1.9.3 I am unable to reproduce this issue. Following is the config I am using and it works without any issue.
var capem = System.IO.File.ReadAllText(caPath);
var certificatepem = System.IO.File.ReadAllText(certPath);
var keypem = System.IO.File.ReadAllText(keyPath);
var adminClient = new AdminClientBuilder(new AdminClientConfig
{
BootstrapServers = bootstrapServers,
SecurityProtocol = SecurityProtocol.Ssl,
SslCaPem = capem,
SslCertificatePem = certificatepem,
SslKeyPem = keypem,
SslKeyPassword = password,
Debug = "all"
}).Build();
@lakshmisivareddy @tittoto please confirm if the previous comment has helped. If so we can close this issue
Description
I have a working config which uses certificates in file path (parameters SslCertificateLocation, SslKeyLocation) and I would like to switch to in-memory certificates instead. When I try to do this, I get the following error:
ssl://_kafka_url_/bootstrap: SSL handshake failed: error:1408E0F4:SSL routines:ssl3_get_message:unexpected message: client SSL authentication might be required (see ssl.key.location and ssl.certificate.location and consult the broker logs for more information) (after 130ms in state SSL_HANDSHAKE)
My CRT file contains the Intermediate certificate authority and it has the following format:
My corresponding KEY file has the following format:
How to reproduce
Working example (with file path):
Not working example:
I took the content of my CRT and KEY files, put them into two variables and use the parameters SslCertificatePem and SslKeyPem. This example is not working.
I did several tests and I found that it also works with the couple SslKeyPem, SslCertificateLocation parameters. So, it seems the issue is only with the SslCertificatePem parameter and I guess it is related to the Intermediate Certificate Authority wich is inside my CRT file.
But what I don't understand is why it works with the file path and why it doesn't work with the in-memory version.
Checklist
Please provide the following information: