search

confluentinc / confluent-kafka-python

Confluent's Kafka Python Client
http://docs.confluent.io/current/clients/confluent-kafka-python
Other
123 stars 896 forks source link

Broker certificate could not be verified. #1610

Open sbv-csis opened 1 year ago

sbv-csis commented 1 year ago

Description

When using openssl s_client I can get the connection to be verified - but when using confluent_kafka it keeps telling me to set a variable I've already set:

Works:


% openssl s_client -connect redpanda-0.redpanda.redpanda.svc.cluster.local:9093  -CAfile /tmp/ca.crt -verify_hostname redpanda-0.redpanda.redpanda.svc.cluster.local
CONNECTED(00000003)
depth=1 CN = redpanda-default-root-certificate
verify return:1
depth=0 
verify return:1
---
Certificate chain
 0 s:
   i:CN = redpanda-default-root-certificate
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Jul 12 10:18:16 2023 GMT; NotAfter: Jul 10 10:18:16 2028 GMT
---
Server certificate
.....
subject=
issuer=CN = redpanda-default-root-certificate
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 1302 bytes and written 801 bytes
Verification: OK
Verified peername: *.redpanda.redpanda.svc.cluster.local
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

But when trying to use confluent-kafka to produce:

>>> from confluent_kafka import Producer
>>> c = {'bootstrap.servers': 'redpanda-0.redpanda.redpanda.svc.cluster.local:9093,redpanda-1.redpanda.redpanda.svc.cluster.local:9093,redpanda-2.redpanda.redpanda.svc.cluster.local:9093', 'security.protocol': 'SASL_SSL', 'sasl.mechanisms': 'SCRAM-SHA-256', 'sasl.username': 'therightusernamehere', 'sasl.password': 'therightpwhere', 'ssl.ca.location': '/tmp/ca.crt'}
>>> p = Producer(c)
>>> p.produce('t1', 'test'.encode())
>>> %3|1689252662.299|FAIL|rdkafka#producer-1| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 3ms in state SSL_HANDSHAKE)

And with full debug:

>>> c['debug'] = 'all'
>>> p = Producer(c)
%7|1689252905.302|SASL|rdkafka#producer-2| [thrd:app]: Selected provider SCRAM (builtin) for SASL mechanism SCRAM-SHA-256
%7|1689252905.302|OPENSSL|rdkafka#producer-2| [thrd:app]: Using statically linked OpenSSL version OpenSSL 3.0.8 7 Feb 2023 (0x30000080, librdkafka built with 0x30000080)
%7|1689252905.303|SSL|rdkafka#producer-2| [thrd:app]: Loading CA certificate(s) from file /tmp/ca.crt
%7|1689252905.303|WAKEUPFD|rdkafka#producer-2| [thrd:app]: sasl_ssl://redpanda-0.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabled low-latency ops queue wake-ups
%7|1689252905.303|BROKER|rdkafka#producer-2| [thrd:app]: sasl_ssl://redpanda-0.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Added new broker with NodeId -1
%7|1689252905.303|BRKMAIN|rdkafka#producer-2| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1689252905.304|BRKMAIN|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-0.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-0.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enter main broker thread
%7|1689252905.304|WAKEUPFD|rdkafka#producer-2| [thrd:app]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabled low-latency ops queue wake-ups
%7|1689252905.304|BROKER|rdkafka#producer-2| [thrd:app]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Added new broker with NodeId -1
%7|1689252905.304|WAKEUPFD|rdkafka#producer-2| [thrd:app]: sasl_ssl://redpanda-2.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabled low-latency ops queue wake-ups
%7|1689252905.304|BROKER|rdkafka#producer-2| [thrd:app]: sasl_ssl://redpanda-2.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Added new broker with NodeId -1
%7|1689252905.304|BRKMAIN|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enter main broker thread
%7|1689252905.304|CONNECT|rdkafka#producer-2| [thrd:app]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1689252905.304|BRKMAIN|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-2.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-2.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enter main broker thread
%7|1689252905.304|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received CONNECT op
%7|1689252905.304|INIT|rdkafka#producer-2| [thrd:app]: librdkafka v2.1.1 (0x20101ff) rdkafka#producer-2 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer,http,oidc, STRIP STATIC_LINKING GCC GXX PKGCONFIG INSTALL GNULD LIBDL PLUGINS ZLIB SSL ZSTD CURL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER OAUTHBEARER_OIDC CRC32C_HW, debug 0xfffff)
%7|1689252905.304|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]: Client configuration:
%7|1689252905.304|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: Broadcasting state change
%7|1689252905.304|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: broker in state TRY_CONNECT connecting
%7|1689252905.304|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1689252905.304|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: Broadcasting state change
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   client.software.name = confluent-kafka-python
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   client.software.version = 2.1.1-rdkafka-2.1.1
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   metadata.broker.list = redpanda-0.redpanda.redpanda.svc.cluster.local:9093,redpanda-1.redpanda.redpanda.svc.cluster.local:9093,redpanda-2.redpanda.redpanda.svc.cluster.local:9093
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   debug = generic,broker,topic,metadata,feature,queue,msg,protocol,cgrp,security,fetch,interceptor,plugin,consumer,admin,eos,mock,assignor,conf,all
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   error_cb = 0x7f6b383e3860
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   opaque = 0x7f6b3904fa60
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   security.protocol = sasl_ssl
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   ssl.ca.location = /tmp/ca.crt
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   sasl.mechanisms = SCRAM-SHA-256
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   sasl.username = [redacted]
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   sasl.password = [redacted]
%7|1689252905.304|CONF|rdkafka#producer-2| [thrd:app]:   dr_msg_cb = 0x7f6b383e12c0
%4|1689252905.304|TERMINATE|rdkafka#producer-1| [thrd:app]: Producer terminating with 1 message (4 bytes) still in queue or transit: use flush() to wait for outstanding message delivery
%7|1689252905.305|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Connecting to ipv4#10.244.13.5:9093 (sasl_ssl) with socket 3
%7|1689252905.305|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Connected to ipv4#10.244.13.5:9093
%7|1689252905.305|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker changed state CONNECT -> SSL_HANDSHAKE
%7|1689252905.305|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: Broadcasting state change
%7|1689252905.305|ENDPOINT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabled endpoint identification using hostname redpanda-1.redpanda.redpanda.svc.cluster.local
>>> %7|1689252905.308|SSLVERIFY|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker SSL certificate verified
%7|1689252905.308|CONNECTED|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Connected (#1)
%7|1689252905.308|FEATURE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1689252905.308|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker changed state SSL_HANDSHAKE -> APIVERSION_QUERY
%7|1689252905.308|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: Broadcasting state change
%7|1689252905.308|SEND|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Sent ApiVersionRequest (v3, 66 bytes @ 0, CorrId 1)
%7|1689252905.350|RECV|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received ApiVersionResponse (v3, 281 bytes, CorrId 1, rtt 41.06ms)
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker API support:
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey Produce (0) Versions 0..7
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey Fetch (1) Versions 4..11
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey ListOffsets (2) Versions 0..4
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey Metadata (3) Versions 0..7
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey OffsetFetch (9) Versions 1..7
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey OffsetDeleteRequest (47) Versions 0..0
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey FindCoordinator (10) Versions 0..3
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey ListGroups (16) Versions 0..3
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey ApiVersion (18) Versions 0..3
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey JoinGroup (11) Versions 0..6
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey Heartbeat (12) Versions 0..4
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey LeaveGroup (13) Versions 0..4
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey SyncGroup (14) Versions 0..4
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey CreateTopics (19) Versions 0..6
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey OffsetCommit (8) Versions 1..8
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DescribeConfigs (32) Versions 0..4
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey AlterConfigs (33) Versions 0..2
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DeleteTopics (20) Versions 0..5
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DescribeGroups (15) Versions 0..5
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey SaslHandshake (17) Versions 0..1
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey SaslAuthenticate (36) Versions 0..2
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey IncrementalAlterConfigsRequest (44) Versions 0..1
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DeleteGroups (42) Versions 0..2
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DescribeAcls (29) Versions 0..2
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DescribeLogDirs (35) Versions 0..2
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey CreateAcls (30) Versions 0..2
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DeleteAcls (31) Versions 0..2
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey InitProducerId (22) Versions 0..3
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey AddPartitionsToTxn (24) Versions 0..3
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey TxnOffsetCommit (28) Versions 0..3
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey AddOffsetsToTxn (25) Versions 0..1
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey EndTxn (26) Versions 0..3
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey CreatePartitions (37) Versions 0..3
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey OffsetForLeaderEpoch (23) Versions 0..4
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey AlterPartitionReassignmentsRequest (45) Versions 0..0
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey ListPartitionReassignmentsRequest (46) Versions 0..0
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DescribeProducers (61) Versions 0..0
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey DescribeTransactions (65) Versions 0..0
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   ApiKey ListTransactions (66) Versions 0..0
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature MsgVer1: Produce (2..2) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature MsgVer1: Fetch (2..2) NOT supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Disabling feature MsgVer1
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature MsgVer2: Produce (3..3) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature MsgVer2: Fetch (4..4) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature MsgVer2
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature ApiVersion: ApiVersion (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature ApiVersion
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature BrokerGroupCoordinator: FindCoordinator (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature BrokerGroupCoordinator
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature BrokerBalancedConsumer: FindCoordinator (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature BrokerBalancedConsumer: OffsetCommit (1..2) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature BrokerBalancedConsumer: OffsetFetch (1..1) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature BrokerBalancedConsumer: JoinGroup (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature BrokerBalancedConsumer: SyncGroup (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature BrokerBalancedConsumer: Heartbeat (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature BrokerBalancedConsumer: LeaveGroup (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature BrokerBalancedConsumer
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature ThrottleTime: Produce (1..2) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature ThrottleTime: Fetch (1..2) NOT supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Disabling feature ThrottleTime
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature Sasl: JoinGroup (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature Sasl
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature SaslHandshake: SaslHandshake (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature SaslHandshake
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature LZ4: FindCoordinator (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature LZ4
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature OffsetTime: ListOffsets (1..1) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature OffsetTime
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature IdempotentProducer: InitProducerId (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature IdempotentProducer
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature ZSTD: Produce (7..7) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature ZSTD: Fetch (10..10) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature ZSTD
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature SaslAuthReq: SaslHandshake (1..1) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:  Feature SaslAuthReq: SaslAuthenticate (0..0) supported by broker
%7|1689252905.350|APIVERSION|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Enabling feature SaslAuthReq
%7|1689252905.350|FEATURE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Updated enabled protocol features to ApiVersion,BrokerBalancedConsumer,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq
%7|1689252905.350|AUTH|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Auth in state APIVERSION_QUERY (handshake supported)
%7|1689252905.350|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker changed state APIVERSION_QUERY -> AUTH_HANDSHAKE
%7|1689252905.351|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: Broadcasting state change
%7|1689252905.351|SEND|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Sent SaslHandshakeRequest (v1, 36 bytes @ 0, CorrId 2)
%7|1689252905.351|RECV|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received SaslHandshakeResponse (v1, 36 bytes, CorrId 2, rtt 0.34ms)
%7|1689252905.351|SASLMECHS|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker supported SASL mechanisms: SCRAM-SHA-256,SCRAM-SHA-512
%7|1689252905.351|AUTH|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Auth in state AUTH_HANDSHAKE (handshake supported)
%7|1689252905.351|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker changed state AUTH_HANDSHAKE -> AUTH_REQ
%7|1689252905.351|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: Broadcasting state change
%7|1689252905.351|SASL|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Initializing SASL client: service name kafka, hostname redpanda-1.redpanda.redpanda.svc.cluster.local, mechanisms SCRAM-SHA-256, provider SCRAM (builtin)
%7|1689252905.351|SASLSCRAM|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: SASL SCRAM client in state client-first-message
%7|1689252905.351|SASL|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Send SASL Kafka frame to broker (43 bytes)
%7|1689252905.351|SEND|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Sent SaslAuthenticateRequest (v0, 68 bytes @ 0, CorrId 3)
%7|1689252905.351|RECV|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received SaslAuthenticateResponse (v0, 358 bytes, CorrId 3, rtt 0.25ms)
%7|1689252905.351|SASL|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received SASL frame from broker (350 bytes)
%7|1689252905.351|SASLSCRAM|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: SASL SCRAM client in state server-first-message
%7|1689252905.379|SASL|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Send SASL Kafka frame to broker (250 bytes)
%7|1689252905.379|SEND|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Sent SaslAuthenticateRequest (v0, 275 bytes @ 0, CorrId 4)
%7|1689252905.420|RECV|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received SaslAuthenticateResponse (v0, 54 bytes, CorrId 4, rtt 40.99ms)
%7|1689252905.420|SASL|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received SASL frame from broker (46 bytes)
%7|1689252905.420|SASLSCRAM|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: SASL SCRAM client in state client-final-message
%7|1689252905.420|SCRAMAUTH|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: SASL SCRAM authentication successful on server: verifying ServerSignature
%7|1689252905.420|SCRAMAUTH|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Authenticated as therealusername using SCRAM-SHA-256
%7|1689252905.420|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Broker changed state AUTH_REQ -> UP
%7|1689252905.420|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: Broadcasting state change
%7|1689252905.420|METADATA|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Request metadata for brokers only: connected
%7|1689252905.420|SEND|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Sent MetadataRequest (v7, 26 bytes @ 0, CorrId 5)
%7|1689252905.420|RECV|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received MetadataResponse (v7, 240 bytes, CorrId 5, rtt 0.39ms)
%7|1689252905.420|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: ===== Received metadata: connected =====
%7|1689252905.420|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: ClusterId: redpanda.9dc56407-546e-44ca-83ed-f45b2aabefb9, ControllerId: 0
%7|1689252905.420|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: 3 brokers, 0 topics
%7|1689252905.420|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   Broker #0/3: redpanda-0.redpanda.redpanda.svc.cluster.local.:9093 NodeId 0
%7|1689252905.420|WAKEUPFD|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-0.redpanda.redpanda.svc.cluster.local.:9093/0: Enabled low-latency ops queue wake-ups
%7|1689252905.420|BROKER|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-0.redpanda.redpanda.svc.cluster.local.:9093/0: Added new broker with NodeId 0
%7|1689252905.421|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   Broker #1/3: redpanda-1.redpanda.redpanda.svc.cluster.local.:9093 NodeId 1
%7|1689252905.421|WAKEUPFD|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Enabled low-latency ops queue wake-ups
%7|1689252905.421|BRKMAIN|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-0.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-0.redpanda.redpanda.svc.cluster.local.:9093/0: Enter main broker thread
%7|1689252905.421|BROKER|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Added new broker with NodeId 1
%7|1689252905.421|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   Broker #2/3: redpanda-2.redpanda.redpanda.svc.cluster.local.:9093 NodeId 2
%7|1689252905.421|WAKEUPFD|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-2.redpanda.redpanda.svc.cluster.local.:9093/2: Enabled low-latency ops queue wake-ups
%7|1689252905.421|BRKMAIN|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Enter main broker thread
%7|1689252905.421|BROKER|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-2.redpanda.redpanda.svc.cluster.local.:9093/2: Added new broker with NodeId 2
%7|1689252905.421|CLUSTERID|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: ClusterId update "" -> "redpanda.9dc56407-546e-44ca-83ed-f45b2aabefb9"
%7|1689252905.421|CONTROLLERID|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: ControllerId update -1 -> 0
%7|1689252905.421|BROADCAST|rdkafka#producer-2| [thrd:main]: Broadcasting state change
%7|1689252905.421|BRKMAIN|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-2.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-2.redpanda.redpanda.svc.cluster.local.:9093/2: Enter main broker thread

>>> p.produce('t1', 'test'.encode())
%7|1689252908.620|TOPIC|rdkafka#producer-2| [thrd:app]: New local topic: t1
%7|1689252908.620|TOPPARNEW|rdkafka#producer-2| [thrd:app]: NEW t1 [-1] 0x561366fb9340 refcnt 0x561366fb93d0 (at rd_kafka_topic_new0:471)
>>> %7|1689252909.304|NOINFO|rdkafka#producer-2| [thrd:main]: Topic t1 metadata information unknown
%7|1689252909.304|NOINFO|rdkafka#producer-2| [thrd:main]: Topic t1 partition count is zero: should refresh metadata
%7|1689252909.304|METADATA|rdkafka#producer-2| [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
%7|1689252909.304|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Request metadata for 1 topic(s): refresh unavailable topics
%7|1689252909.304|SEND|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Sent MetadataRequest (v7, 30 bytes @ 0, CorrId 6)
%7|1689252909.305|RECV|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: Received MetadataResponse (v7, 285 bytes, CorrId 6, rtt 0.35ms)
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: ClusterId: redpanda.9dc56407-546e-44ca-83ed-f45b2aabefb9, ControllerId: 0
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: 3 brokers, 1 topics
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   Topic t1 with 1 partitions
%7|1689252909.305|STATE|rdkafka#producer-2| [thrd:main]: Topic t1 changed state unknown -> exists
%7|1689252909.305|PARTCNT|rdkafka#producer-2| [thrd:main]: Topic t1 partition count changed from 0 to 1
%7|1689252909.305|TOPPARNEW|rdkafka#producer-2| [thrd:main]: NEW t1 [0] 0x7f6b08007c40 refcnt 0x7f6b08007cd0 (at rd_kafka_topic_partition_cnt_update:908)
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]:   Topic t1 partition 0 Leader 1 Epoch -1
%7|1689252909.305|BRKDELGT|rdkafka#producer-2| [thrd:main]: t1 [0]: delegate to broker sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1 (rktp 0x7f6b08007c40, term 0, ref 2)
%7|1689252909.305|BRKDELGT|rdkafka#producer-2| [thrd:main]: t1 [0]: delegating to broker sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1 for partition with 0 messages (0 bytes) queued
%7|1689252909.305|BRKMIGR|rdkafka#producer-2| [thrd:main]: Migrating topic t1 [0] 0x7f6b08007c40 from (none) to sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1 (sending PARTITION_JOIN to sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1)
%7|1689252909.305|PARTCNT|rdkafka#producer-2| [thrd:main]: Partitioning 1 unassigned messages in topic t1 to 1 partitions
%7|1689252909.305|PARTITIONER|rdkafka#producer-2| [thrd:main]: t1 [0] is the new sticky partition
%7|1689252909.305|TOPBRK|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Topic t1 [0]: joining broker (rktp 0x7f6b08007c40, 0 message(s) queued)
%7|1689252909.305|FETCHADD|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Added t1 [0] to active list (1 entries, opv 0, 0 messages queued): joining
%7|1689252909.305|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: Broadcasting state change
%7|1689252909.305|UAS|rdkafka#producer-2| [thrd:main]: 1/1 messages were partitioned in topic t1
%7|1689252909.305|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) in xmit queue (1 added from partition queue)
%7|1689252909.305|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) queued but broker not up
%7|1689252909.305|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Broker changed state INIT -> TRY_CONNECT
%7|1689252909.305|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: Broadcasting state change
%7|1689252909.305|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: broker in state TRY_CONNECT connecting
%7|1689252909.305|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Broker changed state TRY_CONNECT -> CONNECT
%7|1689252909.305|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: Broadcasting state change
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   Broker #0/3: redpanda-0.redpanda.redpanda.svc.cluster.local.:9093 NodeId 0
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   Broker #1/3: redpanda-1.redpanda.redpanda.svc.cluster.local.:9093 NodeId 1
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap:   Broker #2/3: redpanda-2.redpanda.redpanda.svc.cluster.local.:9093 NodeId 2
%7|1689252909.305|METADATA|rdkafka#producer-2| [thrd:main]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local:9093/bootstrap: 1/1 requested topic(s) seen in metadata
%7|1689252909.306|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Connecting to ipv4#10.244.13.5:9093 (sasl_ssl) with socket 10
%7|1689252909.306|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) in xmit queue (0 added from partition queue)
%7|1689252909.306|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) queued but broker not up
%7|1689252909.306|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) in xmit queue (0 added from partition queue)
%7|1689252909.306|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) queued but broker not up
%7|1689252909.306|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Connected to ipv4#10.244.13.5:9093
%7|1689252909.306|STATE|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Broker changed state CONNECT -> SSL_HANDSHAKE
%7|1689252909.306|BROADCAST|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: Broadcasting state change
%7|1689252909.306|ENDPOINT|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: Enabled endpoint identification using hostname redpanda-1.redpanda.redpanda.svc.cluster.local.
%7|1689252909.307|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) in xmit queue (0 added from partition queue)
%7|1689252909.307|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) queued but broker not up
%7|1689252909.307|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) in xmit queue (0 added from partition queue)
%7|1689252909.307|TOPPAR|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: t1 [0] 1 message(s) queued but broker not up
%7|1689252909.309|FAIL|rdkafka#producer-2| [thrd:sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093]: sasl_ssl://redpanda-1.redpanda.redpanda.svc.cluster.local.:9093/1: SSL handshake failed: ssl/statem/statem_clnt.c:1890:tls_post_process_server_certificate error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 3ms in state SSL_HANDSHAKE) (_SSL)

Checklist

Please provide the following information:

>>> import confluent_kafka
>>> confluent_kafka.version()
('2.1.1', 33620224)
>>> confluent_kafka.libversion()
('2.1.1', 33620479)
sbv-csis commented 1 year ago

Sounds like it could be related to https://github.com/confluentinc/librdkafka/issues/4348