search

confluentinc / librdkafka

The Apache Kafka C/C++ library
Other
303 stars 3.15k forks source link

Unable to use sasl_ssl with Python client #4837

Closed abhimanyuseth-alation closed 2 months ago

abhimanyuseth-alation commented 2 months ago

Read the FAQ first: https://github.com/confluentinc/librdkafka/wiki/FAQ

Do NOT create issues for questions, use the discussion forum: https://github.com/confluentinc/librdkafka/discussions

Description

I've configured kafka (3.6.1) with ssl using Confluent documentation https://docs.confluent.io/platform/current/security/security_tutorial.html#generating-keys-certs

I'm able to use the kafka-console-producer.sh and kafka-console-consumer.sh with the jaas configuration to produce and consume a message.

We have python client which uses librdkafka. I'm on the latest confluent-kafka-python library (2.5.0), and ca-certificates package installed. Tried this on ubuntu 20 and RHEL8

Just trying the AdminClient right now and it always fails with certificate error complaining about ssl.ca.location or ca-certificates package. I generated a pem cert as per instructions on librdkafka doc: https://github.com/confluentinc/librdkafka/wiki/Using-SSL-with-librdkafka and passing that for ssl.ca.location

%3|1725429216.481|FAIL|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1889:tls_post_process_server_certificate error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 39ms in state SSL_HANDSHAKE)
%7|1725429216.433|SASL|rdkafka#producer-1| [thrd:app]: Selected provider PLAIN (builtin) for SASL mechanism PLAIN
%7|1725429216.433|OPENSSL|rdkafka#producer-1| [thrd:app]: Using statically linked OpenSSL version OpenSSL 3.0.13 30 Jan 2024 (0x300000d0, librdkafka built with 0x300000d0)
%7|1725429216.439|SSL|rdkafka#producer-1| [thrd:app]: Loading CA certificate(s) from file /var/lib/event_bus/kafka-ssl/librdkafka_client.pem
%7|1725429216.440|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1725429216.440|WAKEUPFD|rdkafka#producer-1| [thrd:app]: sasl_ssl://localhost:9094/bootstrap: Enabled low-latency ops queue wake-ups
%7|1725429216.440|BROKER|rdkafka#producer-1| [thrd:app]: sasl_ssl://localhost:9094/bootstrap: Added new broker with NodeId -1
%7|1725429216.440|CONNECT|rdkafka#producer-1| [thrd:app]: sasl_ssl://localhost:9094/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1725429216.440|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v2.5.0 (0x20500ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer,http,oidc, STRIP STATIC_LINKING GCC GXX PKGCONFIG INSTALL GNULD LDS C11THREADS LIBDL PLUGINS ZLIB SSL ZSTD CURL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER OAUTHBEARER_OIDC CRC32C_HW, debug 0xfffff)
%7|1725429216.440|BRKMAIN|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Enter main broker thread
%7|1725429216.440|CONNECT|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Received CONNECT op
%7|1725429216.440|CONF|rdkafka#producer-1| [thrd:app]: Client configuration:
%7|1725429216.440|STATE|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1725429216.441|BROADCAST|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: Broadcasting state change
%7|1725429216.441|CONNECT|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: broker in state TRY_CONNECT connecting
%7|1725429216.441|STATE|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1725429216.440|CONF|rdkafka#producer-1| [thrd:app]:   client.software.name = confluent-kafka-python
%7|1725429216.441|BROADCAST|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: Broadcasting state change
%7|1725429216.441|CONF|rdkafka#producer-1| [thrd:app]:   client.software.version = 2.5.0-rdkafka-2.5.0
%7|1725429216.441|CONF|rdkafka#producer-1| [thrd:app]:   metadata.broker.list = localhost:9094
%7|1725429216.441|CONF|rdkafka#producer-1| [thrd:app]:   debug = generic,broker,topic,metadata,feature,queue,msg,protocol,cgrp,security,fetch,interceptor,plugin,consumer,admin,eos,mock,assignor,conf,telemetry,all
%7|1725429216.441|CONF|rdkafka#producer-1| [thrd:app]:   error_cb = 0x7ffa3111a330
%7|1725429216.441|CONF|rdkafka#producer-1| [thrd:app]:   background_event_cb = 0x7ffa3110b540
%7|1725429216.441|CONF|rdkafka#producer-1| [thrd:app]:   opaque = 0x7ffa1c5b4670
%7|1725429216.441|CONF|rdkafka#producer-1| [thrd:app]:   security.protocol = sasl_ssl
%7|1725429216.441|CONF|rdkafka#producer-1| [thrd:app]:   ssl.ca.location = /var/lib/event_bus/kafka-ssl/librdkafka_client.pem
%7|1725429216.441|CONF|rdkafka#producer-1| [thrd:app]:   ssl.endpoint.identification.algorithm = none
%7|1725429216.441|CONF|rdkafka#producer-1| [thrd:app]:   sasl.mechanisms = PLAIN
%7|1725429216.441|CONF|rdkafka#producer-1| [thrd:app]:   sasl.username = [redacted]
%7|1725429216.441|CONF|rdkafka#producer-1| [thrd:app]:   sasl.password = [redacted]
%7|1725429216.441|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1725429216.441|CONNECT|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Connecting to ipv6#[::1]:9094 (sasl_ssl) with socket 17
%7|1725429216.441|CONNECT|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Connected to ipv6#[::1]:9094
%7|1725429216.441|STATE|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Broker changed state CONNECT -> SSL_HANDSHAKE
%7|1725429216.441|BROADCAST|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: Broadcasting state change
%7|1725429216.442|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 48ms: application metadata request
%7|1725429216.481|FAIL|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1889:tls_post_process_server_certificate error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 39ms in state SSL_HANDSHAKE) (_SSL)
%3|1725429216.481|FAIL|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1889:tls_post_process_server_certificate error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 39ms in state SSL_HANDSHAKE)
%7|1725429216.481|STATE|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Broker changed state SSL_HANDSHAKE -> DOWN
%7|1725429216.481|BROADCAST|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: Broadcasting state change
%7|1725429216.481|BUFQ|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Purging bufq with 0 buffers
%7|1725429216.481|BUFQ|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Purging bufq with 0 buffers
%7|1725429216.481|BUFQ|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Updating 0 buffers on connection reset
%7|1725429216.481|STATE|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Broker changed state DOWN -> INIT
%7|1725429216.481|BROADCAST|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: Broadcasting state change
%7|1725429216.481|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 9ms: application metadata request
%7|1725429217.440|CONNECT|rdkafka#producer-1| [thrd:main]: sasl_ssl://localhost:9094/bootstrap: Selected for cluster connection: no cluster connection (broker has 1 connection attempt(s))
%7|1725429217.440|CONNECT|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Received CONNECT op
%7|1725429217.440|STATE|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1725429217.440|BROADCAST|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: Broadcasting state change
%7|1725429217.440|CONNECT|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: broker in state TRY_CONNECT connecting
%7|1725429217.440|STATE|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1725429217.440|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1725429217.440|BROADCAST|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: Broadcasting state change
%7|1725429217.440|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1725429217.441|CONNECT|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Connecting to ipv4#127.0.0.1:9094 (sasl_ssl) with socket 17
%7|1725429217.441|CONNECT|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Connected to ipv4#127.0.0.1:9094
%7|1725429217.441|STATE|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: Broker changed state CONNECT -> SSL_HANDSHAKE
%7|1725429217.441|BROADCAST|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: Broadcasting state change
%7|1725429217.441|CONNECT|rdkafka#producer-1| [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
%7|1725429217.451|FAIL|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1889:tls_post_process_server_certificate error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 10ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed) (_SSL): identical to last error
%3|1725429217.451|FAIL|rdkafka#producer-1| [thrd:sasl_ssl://localhost:9094/bootstrap]: sasl_ssl://localhost:9094/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1889:tls_post_process_server_certificate error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 10ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)

How to reproduce

Setup Kafka and Zookeeper as per Confluent SSL doc https://docs.confluent.io/platform/current/security/security_tutorial.html#generating-keys-certs

Setup latest librdkafka 2.5 and install confluent-kafka-python 2.5.0

admin_config["security.protocol"] = "SASL_SSL"
admin_config["sasl.mechanism"] = "PLAIN"
ssl_dir = conf["event_bus.kafka.ssl.conf_path"]
admin_config["ssl.ca.location"] = f"{ssl_dir}/librdkafka_client.pem"
admin_config["ssl.endpoint.identification.algorithm"] = "none"
admin_config["sasl.username"] = conf["event_bus.kafka.ssl.sasl.admin_user"]
admin_config["sasl.password"] = conf.get_decrypted(
            "event_bus.kafka.ssl.sasl.admin_password"
        )
admin_config["debug"] = "all"
admin_config["bootstrap.servers"] = get_bootstrap_server()
client = AdminClient(admin_config)
md = client.list_topics(timeout=timeout)

IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/confluentinc/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

abhimanyuseth-alation commented 2 months ago

I even tried passing cacert.pem but got the same error.

abhimanyuseth-alation commented 2 months ago

So I found a post that mentioned converting the jks keystore to a pem cert https://stackoverflow.com/a/656559/4484956

Using this, I converted the jks keystore used in the kafka broker to a pem cert and specified the pem file in ssl.ca.location and it worked!