Serg675 commented 3 years ago

Description

Hello, I need to produce to Kafka on OpenShift/OKD cluster deployed using https://strimzi.io. The specific of this setup is that Kafka brokers are not accessible from outside of the cluster. Kafka can be accessed via route service only which is TLS encrypted. I try to access Kafka using 1.5.3 version of the client on Windows 10 and Windows Server 2019. I use information from this source https://blog.mimacom.com/strimzi-okd/ from "External Access of Kafka" chapter. I have extracted ca.crt as described in mentioned article. The problem is that this certificate has public key only. No private key. Java client and Kafka consumer and producer console scripts work fine with .JKS trusted store created with this exported certificate (as depicted in the article). But Kafka .NET client does not. I use following setting:

<add key="bootstrap.servers" value="kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxx.com:443"/> <add key="ssl.endpoint.identification.algorithm" value="none" /> <add key="security.protocol" value="SSL" /> <add key="ssl.certificate.location" value="ca.crt" /> <add key="ssl.key.password" value="password" />

I read these settings from the app.config into Dictionary<string,string> and feed it to ProducerBuilder. Works fine for all other setting (not SSL related). So I'm sure that settings are adopted by ProducerBuilder.

I get SSL handshake failed error.

%3|1608580696.853|FAIL|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxx.com:443/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 479ms in state CONNECT) %3|1608580696.853|ERROR|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxx]: 1/1 brokers are down

Could you please answer below questions?

Is private key mandatory to configure SSL?
If not then what is right configuration to use public key only?

Thank you in advance.

Adding console output with debug=protocol,security,broker

%7|1608583003.853|OPENSSL|rdkafka#producer-1| [thrd:app]: librdkafka built with OpenSSL version 0x1000211f %7|1608583004.287|CERTROOT|rdkafka#producer-1| [thrd:app]: 74/74 certificate(s) successfully added from Windows Certificate Root store %7|1608583004.287|SSL|rdkafka#producer-1| [thrd:app]: Loading public key from file ca.crt %7|1608583004.290|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread %7|1608583004.290|BROKER|rdkafka#producer-1| [thrd:app]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Added new broker with NodeId -1 %7|1608583004.290|CONNECT|rdkafka#producer-1| [thrd:app]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s)) %7|1608583004.290|BRKMAIN|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Enter main broker thread %7|1608583004.290|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.5.3 (0x10503ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, SSL ZLIB SNAPPY SASL_SCRAM PLUGINS HDRHISTOGRAM, debug 0x282) %7|1608583004.290|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Received CONNECT op %7|1608583004.291|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Broker changed state INIT -> TRY_CONNECT %7|1608583004.291|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: broker in state TRY_CONNECT connecting %7|1608583004.291|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Broker changed state TRY_CONNECT -> CONNECT %7|1608583004.295|OPENSSL|rdkafka#producer-2| [thrd:app]: librdkafka built with OpenSSL version 0x1000211f %7|1608583004.302|CERTROOT|rdkafka#producer-2| [thrd:app]: 74/74 certificate(s) successfully added from Windows Certificate Root store %7|1608583004.304|SSL|rdkafka#producer-2| [thrd:app]: Loading public key from file ca.crt %7|1608583004.306|BRKMAIN|rdkafka#producer-2| [thrd::0/internal]: :0/internal: Enter main broker thread %7|1608583004.306|BROKER|rdkafka#producer-2| [thrd:app]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Added new broker with NodeId -1 %7|1608583004.307|CONNECT|rdkafka#producer-2| [thrd:app]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s)) %7|1608583004.307|BRKMAIN|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Enter main broker thread %7|1608583004.307|INIT|rdkafka#producer-2| [thrd:app]: librdkafka v1.5.3 (0x10503ff) rdkafka#producer-2 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, SSL ZLIB SNAPPY SASL_SCRAM PLUGINS HDRHISTOGRAM, debug 0x282) %7|1608583004.307|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Received CONNECT op %7|1608583004.307|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Broker changed state INIT -> TRY_CONNECT %7|1608583004.308|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: broker in state TRY_CONNECT connecting %7|1608583004.308|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Broker changed state TRY_CONNECT -> CONNECT %7|1608583004.338|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Connecting to ipv4#111.111.111.111:443 (ssl) with socket 1960 %7|1608583004.338|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Connecting to ipv4#111.111.111.111:443 (ssl) with socket 1484 %7|1608583004.575|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Connected to ipv4#111.111.111.111:443 %7|1608583004.586|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Connected to ipv4#111.111.111.111:443 %7|1608583004.815|FAIL|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 524ms in state CONNECT) (_SSL) %3|1608583004.816|FAIL|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 524ms in state CONNECT) %7|1608583004.817|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: Broker changed state CONNECT -> DOWN %3|1608583004.817|ERROR|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcimcluster.sapienc]: 1/1 brokers are down %3|1608583004.817|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxxxxxxxx.com:443/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 524ms in state CONNECT)

Checklist

Please provide the following information:

[ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
[x] Confluent.Kafka nuget version.
[ ] Apache Kafka version.
[x] Client configuration.
[x] Operating system.
[x] Provide logs (with "debug" : "..." as necessary in configuration).
[ ] Provide broker log excerpts.
[x] Critical issue.

Serg675 commented 3 years ago

Hello, I have figured out how to configure. I have used below command to extract all secrets oc extract secret/logs-cluster-ca-cert --to=DIRECTORY It's 3 files ca.crt, ca.p12 and ca.password . I have installed ca.p12 (double click on the file) for Windows Local Machine on the same host when I run .NET client. It will ask for password and password is in ca.password file. I use following setting for .NET client. <add key="ssl.endpoint.identification.algorithm" value="none" /> <add key="security.protocol" value="SSL" /> <add key="ssl.certificate.location" value="ca.crt" /> It works. I can produce to the OpenShift/OKD cluster. But periodically the client gets disconnected and connected back. Looks like an issue to me but don't know how serious it is. Below is the log.

%7|1608650340.523|OPENSSL|rdkafka#producer-1| [thrd:app]: librdkafka built with OpenSSL version 0x1000211f %7|1608650341.012|CERTROOT|rdkafka#producer-1| [thrd:app]: 75/75 certificate(s) successfully added from Windows Certificate Root store %7|1608650341.012|SSL|rdkafka#producer-1| [thrd:app]: Loading public key from file ca.crt %7|1608650341.014|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread %7|1608650341.015|BROKER|rdkafka#producer-1| [thrd:app]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Added new broker with NodeId -1 %7|1608650341.015|CONNECT|rdkafka#producer-1| [thrd:app]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s)) %7|1608650341.015|BRKMAIN|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Enter main broker thread %7|1608650341.015|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Received CONNECT op %7|1608650341.015|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state INIT -> TRY_CONNECT %7|1608650341.015|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: broker in state TRY_CONNECT connecting %7|1608650341.015|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state TRY_CONNECT -> CONNECT %7|1608650341.015|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.5.3 (0x10503ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, SSL ZLIB SNAPPY SASL_SCRAM PLUGINS HDRHISTOGRAM, debug 0x282) %7|1608650341.021|OPENSSL|rdkafka#producer-2| [thrd:app]: librdkafka built with OpenSSL version 0x1000211f %7|1608650341.033|CERTROOT|rdkafka#producer-2| [thrd:app]: 75/75 certificate(s) successfully added from Windows Certificate Root store %7|1608650341.034|SSL|rdkafka#producer-2| [thrd:app]: Loading public key from file ca.crt %7|1608650341.036|BRKMAIN|rdkafka#producer-2| [thrd::0/internal]: :0/internal: Enter main broker thread %7|1608650341.036|BROKER|rdkafka#producer-2| [thrd:app]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Added new broker with NodeId -1 %7|1608650341.037|CONNECT|rdkafka#producer-2| [thrd:app]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s)) %7|1608650341.037|BRKMAIN|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Enter main broker thread %7|1608650341.037|INIT|rdkafka#producer-2| [thrd:app]: librdkafka v1.5.3 (0x10503ff) rdkafka#producer-2 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, SSL ZLIB SNAPPY SASL_SCRAM PLUGINS HDRHISTOGRAM, debug 0x282) %7|1608650341.037|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Received CONNECT op %7|1608650341.037|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state INIT -> TRY_CONNECT %7|1608650341.038|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: broker in state TRY_CONNECT connecting %7|1608650341.038|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state TRY_CONNECT -> CONNECT %7|1608650341.066|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Connecting to ipv4#111.111.111.111:443 (ssl) with socket 1488 %7|1608650341.066|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Connecting to ipv4#111.111.111.111:443 (ssl) with socket 1856 %7|1608650341.315|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Connected to ipv4#111.111.111.111:443 %7|1608650341.326|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Connected to ipv4#111.111.111.111:443 %7|1608650341.841|SSLVERIFY|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker SSL certificate verified %7|1608650341.841|CONNECTED|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Connected (#1) %7|1608650341.843|FEATURE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion %7|1608650341.843|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY %7|1608650341.844|SEND|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Sent ApiVersionRequest (v3, 52 bytes @ 0, CorrId 1) %7|1608650341.853|SSLVERIFY|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker SSL certificate verified %7|1608650341.855|CONNECTED|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Connected (#1) %7|1608650341.855|FEATURE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion %7|1608650341.855|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY %7|1608650341.856|SEND|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Sent ApiVersionRequest (v3, 52 bytes @ 0, CorrId 1) %7|1608650342.021|OPENSSL|rdkafka#producer-3| [thrd:app]: librdkafka built with OpenSSL version 0x1000211f %7|1608650342.023|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: no cluster connection %7|1608650342.031|CERTROOT|rdkafka#producer-3| [thrd:app]: 75/75 certificate(s) successfully added from Windows Certificate Root store %7|1608650342.031|SSL|rdkafka#producer-3| [thrd:app]: Loading public key from file ca.crt %7|1608650342.033|BRKMAIN|rdkafka#producer-3| [thrd::0/internal]: :0/internal: Enter main broker thread %7|1608650342.034|BROKER|rdkafka#producer-3| [thrd:app]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Added new broker with NodeId -1 %7|1608650342.034|CONNECT|rdkafka#producer-3| [thrd:app]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s)) %7|1608650342.034|BRKMAIN|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Enter main broker thread %7|1608650342.034|INIT|rdkafka#producer-3| [thrd:app]: librdkafka v1.5.3 (0x10503ff) rdkafka#producer-3 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, SSL ZLIB SNAPPY SASL_SCRAM PLUGINS HDRHISTOGRAM, debug 0x282) %7|1608650342.035|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Received CONNECT op %7|1608650342.035|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state INIT -> TRY_CONNECT

%7|1608650342.036|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: broker in state TRY_CONNECT connecting %7|1608650342.036|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state TRY_CONNECT -> CONNECT %7|1608650342.037|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Connecting to ipv4#111.111.111.111:443 (ssl) with socket 2428 %7|1608650342.038|CONNECT|rdkafka#producer-2| [thrd:main]: Cluster connection already in progress: no cluster connection %7|1608650342.094|RECV|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Received ApiVersionResponse (v3, 358 bytes, CorrId 1, rtt 249.80ms) %7|1608650342.094|FEATURE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq %7|1608650342.095|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state APIVERSION_QUERY -> UP %7|1608650342.096|SEND|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Sent MetadataRequest (v4, 26 bytes @ 0, CorrId 2) %7|1608650342.105|RECV|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Received ApiVersionResponse (v3, 358 bytes, CorrId 1, rtt 249.41ms) %7|1608650342.106|FEATURE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq %7|1608650342.106|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state APIVERSION_QUERY -> UP %7|1608650342.106|SEND|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Sent MetadataRequest (v4, 26 bytes @ 0, CorrId 2) %7|1608650342.277|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Connected to ipv4#111.111.111.111:443 %7|1608650342.349|RECV|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Received MetadataResponse (v4, 265 bytes, CorrId 2, rtt 252.73ms) %7|1608650342.351|BROKER|rdkafka#producer-2| [thrd:main]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Added new broker with NodeId 0 %7|1608650342.352|BRKMAIN|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Enter main broker thread %7|1608650342.353|BROKER|rdkafka#producer-2| [thrd:main]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Added new broker with NodeId 2 %7|1608650342.353|BRKMAIN|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Enter main broker thread %7|1608650342.354|BROKER|rdkafka#producer-2| [thrd:main]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Added new broker with NodeId 1 %7|1608650342.354|CLUSTERID|rdkafka#producer-2| [thrd:main]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: ClusterId update "" -> "DsNGUYIzQ0GBew20z4eNcw" %7|1608650342.355|CONTROLLERID|rdkafka#producer-2| [thrd:main]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: ControllerId update -1 -> 2 %7|1608650342.354|BRKMAIN|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Enter main broker thread %7|1608650342.356|RECV|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Received MetadataResponse (v4, 265 bytes, CorrId 2, rtt 249.79ms) %7|1608650342.357|BROKER|rdkafka#producer-1| [thrd:main]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Added new broker with NodeId 0 %7|1608650342.358|BRKMAIN|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Enter main broker thread %7|1608650342.358|BROKER|rdkafka#producer-1| [thrd:main]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Added new broker with NodeId 2 %7|1608650342.359|BRKMAIN|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Enter main broker thread %7|1608650342.359|BROKER|rdkafka#producer-1| [thrd:main]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Added new broker with NodeId 1 %7|1608650342.359|CLUSTERID|rdkafka#producer-1| [thrd:main]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: ClusterId update "" -> "DsNGUYIzQ0GBew20z4eNcw" %7|1608650342.360|CONTROLLERID|rdkafka#producer-1| [thrd:main]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: ControllerId update -1 -> 2 %7|1608650342.360|BRKMAIN|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Enter main broker thread %7|1608650342.760|SSLVERIFY|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker SSL certificate verified %7|1608650342.760|CONNECTED|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Connected (#1) %7|1608650342.761|FEATURE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion %7|1608650342.761|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY %7|1608650342.762|SEND|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Sent ApiVersionRequest (v3, 52 bytes @ 0, CorrId 1) %7|1608650342.996|RECV|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Received ApiVersionResponse (v3, 358 bytes, CorrId 1, rtt 233.18ms) %7|1608650342.996|FEATURE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq %7|1608650342.997|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state APIVERSION_QUERY -> UP %7|1608650342.998|SEND|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Sent MetadataRequest (v4, 26 bytes @ 0, CorrId 2) %7|1608650343.232|RECV|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Received MetadataResponse (v4, 265 bytes, CorrId 2, rtt 234.15ms) %7|1608650343.233|BROKER|rdkafka#producer-3| [thrd:main]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Added new broker with NodeId 0 %7|1608650343.233|BRKMAIN|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Enter main broker thread %7|1608650343.234|BROKER|rdkafka#producer-3| [thrd:main]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Added new broker with NodeId 2 %7|1608650343.234|BRKMAIN|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Enter main broker thread %7|1608650343.235|BROKER|rdkafka#producer-3| [thrd:main]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Added new broker with NodeId 1 %7|1608650343.235|CLUSTERID|rdkafka#producer-3| [thrd:main]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: ClusterId update "" -> "DsNGUYIzQ0GBew20z4eNcw" %7|1608650343.236|CONTROLLERID|rdkafka#producer-3| [thrd:main]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: ControllerId update -1 -> 2 %7|1608650343.236|BRKMAIN|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Enter main broker thread %7|1608650401.512|FAIL|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Disconnected (after 59416ms in state UP) (_TRANSPORT) %6|1608650401.513|FAIL|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Disconnected (after 59416ms in state UP) %7|1608650401.514|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state UP -> DOWN %7|1608650401.514|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state DOWN -> INIT %3|1608650401.514|ERROR|rdkafka#producer-2| [thrd:app]: rdkafka#producer-2: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Disconnected (after 59416ms in state UP) %7|1608650401.521|FAIL|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Disconnected (after 59415ms in state UP) (_TRANSPORT) %6|1608650401.522|FAIL|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Disconnected (after 59415ms in state UP) %7|1608650401.522|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state UP -> DOWN %7|1608650401.522|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state DOWN -> INIT %3|1608650401.522|ERROR|rdkafka#producer-1| [thrd:app]: rdkafka#producer-1: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Disconnected (after 59415ms in state UP) %7|1608650402.202|CONNECT|rdkafka#producer-2| [thrd:main]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Selected for cluster connection: no cluster connection (broker has 0 connection attempt(s)) %7|1608650402.202|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Received CONNECT op %7|1608650402.205|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state INIT -> TRY_CONNECT %7|1608650402.205|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: broker in state TRY_CONNECT connecting %7|1608650402.205|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state TRY_CONNECT -> CONNECT %7|1608650402.217|CONNECT|rdkafka#producer-1| [thrd:main]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Selected for cluster connection: no cluster connection (broker has 0 connection attempt(s)) %7|1608650402.220|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Received CONNECT op %7|1608650402.222|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state INIT -> TRY_CONNECT %7|1608650402.222|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: broker in state TRY_CONNECT connecting %7|1608650402.223|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state TRY_CONNECT -> CONNECT %7|1608650402.246|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Connecting to ipv4#222.222.222.222:443 (ssl) with socket 2104 %7|1608650402.246|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Connecting to ipv4#222.222.222.222:443 (ssl) with socket 2036 %7|1608650402.494|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Connected to ipv4#222.222.222.222:443 %7|1608650402.509|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Connected to ipv4#222.222.222.222:443 %7|1608650402.510|FAIL|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Disconnected (after 59513ms in state UP) (_TRANSPORT) %6|1608650402.510|FAIL|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Disconnected (after 59513ms in state UP) %7|1608650402.511|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state UP -> DOWN %3|1608650402.511|ERROR|rdkafka#producer-3| [thrd:app]: rdkafka#producer-3: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Disconnected (after 59513ms in state UP) %7|1608650402.512|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxx]: ssl://kafka-prod-kafka-bootstrap-dev.apps.okdcluster.xxxxxxxxx.com:443/bootstrap: Broker changed state DOWN -> INIT %7|1608650403.002|SSLVERIFY|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker SSL certificate verified %7|1608650403.002|CONNECTED|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Connected (#1) %7|1608650403.003|FEATURE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Updated enabled protocol features +ApiVersion to ApiVersion %7|1608650403.004|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state CONNECT -> APIVERSION_QUERY %7|1608650403.004|SEND|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Sent ApiVersionRequest (v3, 52 bytes @ 0, CorrId 1) %7|1608650403.042|SSLVERIFY|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker SSL certificate verified %7|1608650403.042|CONNECTED|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Connected (#1) %7|1608650403.044|FEATURE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Updated enabled protocol features +ApiVersion to ApiVersion %7|1608650403.044|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state CONNECT -> APIVERSION_QUERY %7|1608650403.044|SEND|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Sent ApiVersionRequest (v3, 52 bytes @ 0, CorrId 1) %7|1608650403.206|CONNECT|rdkafka#producer-2| [thrd:main]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Selected for cluster connection: no cluster connection (broker has 0 connection attempt(s)) %7|1608650403.207|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Received CONNECT op %7|1608650403.209|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state INIT -> TRY_CONNECT %7|1608650403.210|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: broker in state TRY_CONNECT connecting %7|1608650403.210|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state TRY_CONNECT -> CONNECT %7|1608650403.222|CONNECT|rdkafka#producer-1| [thrd:main]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Selected for cluster connection: no cluster connection (broker has 0 connection attempt(s)) %7|1608650403.224|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Received CONNECT op %7|1608650403.225|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state INIT -> TRY_CONNECT %7|1608650403.225|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: broker in state TRY_CONNECT connecting %7|1608650403.226|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state TRY_CONNECT -> CONNECT %7|1608650403.258|RECV|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Received ApiVersionResponse (v3, 358 bytes, CorrId 1, rtt 253.37ms) %7|1608650403.258|FEATURE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq %7|1608650403.259|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state APIVERSION_QUERY -> UP %7|1608650403.260|SEND|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Sent MetadataRequest (v4, 26 bytes @ 0, CorrId 2) %7|1608650403.289|RECV|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Received ApiVersionResponse (v3, 358 bytes, CorrId 1, rtt 244.05ms) %7|1608650403.289|FEATURE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq %7|1608650403.290|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state APIVERSION_QUERY -> UP %7|1608650403.290|SEND|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Sent MetadataRequest (v4, 26 bytes @ 0, CorrId 2) %7|1608650403.291|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Connecting to ipv4#111.111.111.111:443 (ssl) with socket 2116 %7|1608650403.291|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Connecting to ipv4#111.111.111.111:443 (ssl) with socket 1204 %7|1608650403.299|CONNECT|rdkafka#producer-3| [thrd:main]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Selected for cluster connection: no cluster connection (broker has 0 connection attempt(s)) %7|1608650403.300|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Received CONNECT op %7|1608650403.301|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state INIT -> TRY_CONNECT %7|1608650403.302|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: broker in state TRY_CONNECT connecting %7|1608650403.302|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state TRY_CONNECT -> CONNECT %7|1608650403.303|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Connecting to ipv4#222.222.222.222:443 (ssl) with socket 3000 %7|1608650403.498|RECV|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Received MetadataResponse (v4, 265 bytes, CorrId 2, rtt 238.32ms) %7|1608650403.536|RECV|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Received MetadataResponse (v4, 265 bytes, CorrId 2, rtt 245.51ms) %7|1608650403.548|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Connected to ipv4#222.222.222.222:443 %7|1608650403.560|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Connected to ipv4#111.111.111.111:443 %7|1608650403.560|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Connected to ipv4#111.111.111.111:443 %7|1608650404.087|SSLVERIFY|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker SSL certificate verified %7|1608650404.087|CONNECTED|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Connected (#1) %7|1608650404.089|FEATURE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Updated enabled protocol features +ApiVersion to ApiVersion %7|1608650404.089|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state CONNECT -> APIVERSION_QUERY %7|1608650404.090|SEND|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Sent ApiVersionRequest (v3, 52 bytes @ 0, CorrId 1) %7|1608650404.097|SSLVERIFY|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker SSL certificate verified %7|1608650404.097|CONNECTED|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Connected (#1) %7|1608650404.098|FEATURE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Updated enabled protocol features +ApiVersion to ApiVersion %7|1608650404.098|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state CONNECT -> APIVERSION_QUERY %7|1608650404.099|SEND|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Sent ApiVersionRequest (v3, 52 bytes @ 0, CorrId 1) %7|1608650404.130|SSLVERIFY|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker SSL certificate verified %7|1608650404.130|CONNECTED|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Connected (#1) %7|1608650404.131|FEATURE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Updated enabled protocol features +ApiVersion to ApiVersion %7|1608650404.131|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state CONNECT -> APIVERSION_QUERY %7|1608650404.132|SEND|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Sent ApiVersionRequest (v3, 52 bytes @ 0, CorrId 1) %7|1608650404.302|CONNECT|rdkafka#producer-3| [thrd:main]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Selected for cluster connection: no cluster connection (broker has 0 connection attempt(s)) %7|1608650404.302|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Received CONNECT op %7|1608650404.305|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state INIT -> TRY_CONNECT %7|1608650404.306|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: broker in state TRY_CONNECT connecting %7|1608650404.306|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state TRY_CONNECT -> CONNECT %7|1608650404.307|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Connecting to ipv4#111.111.111.111:443 (ssl) with socket 3044 %7|1608650404.329|RECV|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Received ApiVersionResponse (v3, 358 bytes, CorrId 1, rtt 238.65ms) %7|1608650404.329|FEATURE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq %7|1608650404.329|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state APIVERSION_QUERY -> UP %7|1608650404.330|SEND|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Sent MetadataRequest (v4, 26 bytes @ 0, CorrId 2) %7|1608650404.347|RECV|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Received ApiVersionResponse (v3, 358 bytes, CorrId 1, rtt 248.06ms) %7|1608650404.347|FEATURE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq %7|1608650404.348|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state APIVERSION_QUERY -> UP %7|1608650404.350|SEND|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Sent MetadataRequest (v4, 26 bytes @ 0, CorrId 2) %7|1608650404.388|RECV|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Received ApiVersionResponse (v3, 358 bytes, CorrId 1, rtt 256.44ms) %7|1608650404.388|FEATURE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq %7|1608650404.390|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state APIVERSION_QUERY -> UP %7|1608650404.391|SEND|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Sent MetadataRequest (v4, 26 bytes @ 0, CorrId 2) %7|1608650404.569|RECV|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Received MetadataResponse (v4, 265 bytes, CorrId 2, rtt 238.40ms) %7|1608650404.578|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Connected to ipv4#111.111.111.111:443 %7|1608650404.606|RECV|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Received MetadataResponse (v4, 265 bytes, CorrId 2, rtt 255.94ms) %7|1608650404.638|RECV|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Received MetadataResponse (v4, 265 bytes, CorrId 2, rtt 246.95ms) %7|1608650405.094|SSLVERIFY|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker SSL certificate verified %7|1608650405.094|CONNECTED|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Connected (#1) %7|1608650405.095|FEATURE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Updated enabled protocol features +ApiVersion to ApiVersion %7|1608650405.096|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state CONNECT -> APIVERSION_QUERY %7|1608650405.097|SEND|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Sent ApiVersionRequest (v3, 52 bytes @ 0, CorrId 1) %7|1608650405.342|RECV|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Received ApiVersionResponse (v3, 358 bytes, CorrId 1, rtt 245.34ms) %7|1608650405.342|FEATURE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq %7|1608650405.343|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state APIVERSION_QUERY -> UP %7|1608650405.344|SEND|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Sent MetadataRequest (v4, 26 bytes @ 0, CorrId 2) %7|1608650405.594|RECV|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Received MetadataResponse (v4, 265 bytes, CorrId 2, rtt 249.57ms) %7|1608650463.374|FAIL|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Disconnected (after 60114ms in state UP) (_TRANSPORT) %7|1608650463.375|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state UP -> DOWN %7|1608650463.375|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state DOWN -> INIT %7|1608650463.374|FAIL|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Disconnected (after 60084ms in state UP) (_TRANSPORT) %7|1608650463.375|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state UP -> DOWN %7|1608650463.376|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state DOWN -> INIT %7|1608650464.374|FAIL|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Disconnected (after 59983ms in state UP) (_TRANSPORT) %6|1608650464.374|FAIL|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Disconnected (after 59983ms in state UP) %7|1608650464.375|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state UP -> DOWN %3|1608650464.375|ERROR|rdkafka#producer-3| [thrd:app]: rdkafka#producer-3: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Disconnected (after 59983ms in state UP) %7|1608650464.375|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-1-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/1: Broker changed state DOWN -> INIT %7|1608650464.510|FAIL|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Disconnected (after 60159ms in state UP) (_TRANSPORT) %7|1608650464.510|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state UP -> DOWN %7|1608650464.510|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state DOWN -> INIT %7|1608650464.514|FAIL|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Disconnected (after 60184ms in state UP) (_TRANSPORT) %7|1608650464.514|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state UP -> DOWN %7|1608650464.514|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state DOWN -> INIT %7|1608650464.516|CONNECT|rdkafka#producer-1| [thrd:main]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Selected for cluster connection: no cluster connection (broker has 0 connection attempt(s)) %7|1608650464.517|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Received CONNECT op %7|1608650464.519|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Broker changed state INIT -> TRY_CONNECT %7|1608650464.519|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: broker in state TRY_CONNECT connecting %7|1608650464.519|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Broker changed state TRY_CONNECT -> CONNECT %7|1608650464.563|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Connecting to ipv4#222.222.222.222:443 (ssl) with socket 1996 %7|1608650464.824|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Connected to ipv4#222.222.222.222:443 %7|1608650465.324|SSLVERIFY|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Broker SSL certificate verified %7|1608650465.324|CONNECTED|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Connected (#1) %7|1608650465.325|FEATURE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Updated enabled protocol features +ApiVersion to ApiVersion %7|1608650465.326|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Broker changed state CONNECT -> APIVERSION_QUERY %7|1608650465.326|SEND|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Sent ApiVersionRequest (v3, 52 bytes @ 0, CorrId 1) %7|1608650465.489|CONNECT|rdkafka#producer-2| [thrd:main]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Selected for cluster connection: no cluster connection (broker has 0 connection attempt(s)) %7|1608650465.489|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Received CONNECT op %7|1608650465.491|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Broker changed state INIT -> TRY_CONNECT %7|1608650465.492|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: broker in state TRY_CONNECT connecting %7|1608650465.492|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Broker changed state TRY_CONNECT -> CONNECT %7|1608650465.493|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Connecting to ipv4#222.222.222.222:443 (ssl) with socket 2132 %7|1608650465.511|FAIL|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Disconnected (after 60167ms in state UP) (_TRANSPORT) %7|1608650465.511|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state UP -> DOWN %7|1608650465.512|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state DOWN -> INIT %7|1608650465.520|CONNECT|rdkafka#producer-1| [thrd:main]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Selected for cluster connection: no cluster connection (broker has 1 connection attempt(s)) %7|1608650465.521|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Received CONNECT op %7|1608650465.523|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state INIT -> TRY_CONNECT %7|1608650465.523|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: broker in state TRY_CONNECT connecting %7|1608650465.523|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state TRY_CONNECT -> CONNECT %7|1608650465.564|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Connecting to ipv4#222.222.222.222:443 (ssl) with socket 2088 %7|1608650465.568|RECV|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Received ApiVersionResponse (v3, 358 bytes, CorrId 1, rtt 241.53ms) %7|1608650465.568|FEATURE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq %7|1608650465.568|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Broker changed state APIVERSION_QUERY -> UP %7|1608650465.569|SEND|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Sent MetadataRequest (v4, 26 bytes @ 0, CorrId 2) %7|1608650465.581|CONNECT|rdkafka#producer-3| [thrd:main]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Selected for cluster connection: no cluster connection (broker has 0 connection attempt(s)) %7|1608650465.583|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Received CONNECT op %7|1608650465.583|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Broker changed state INIT -> TRY_CONNECT %7|1608650465.584|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: broker in state TRY_CONNECT connecting %7|1608650465.584|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Broker changed state TRY_CONNECT -> CONNECT %7|1608650465.585|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Connecting to ipv4#222.222.222.222:443 (ssl) with socket 2140 %7|1608650465.750|CONNECT|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Connected to ipv4#222.222.222.222:443 %7|1608650465.813|RECV|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Received MetadataResponse (v4, 265 bytes, CorrId 2, rtt 243.79ms) %7|1608650465.828|CONNECT|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Connected to ipv4#222.222.222.222:443 %7|1608650465.844|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Connected to ipv4#222.222.222.222:443 %7|1608650466.233|SSLVERIFY|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Broker SSL certificate verified %7|1608650466.233|CONNECTED|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Connected (#1) %7|1608650466.234|FEATURE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Updated enabled protocol features +ApiVersion to ApiVersion %7|1608650466.235|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Broker changed state CONNECT -> APIVERSION_QUERY %7|1608650466.236|SEND|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Sent ApiVersionRequest (v3, 52 bytes @ 0, CorrId 1) %7|1608650466.326|SSLVERIFY|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker SSL certificate verified %7|1608650466.327|CONNECTED|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Connected (#2) %7|1608650466.328|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state CONNECT -> APIVERSION_QUERY %7|1608650466.329|SEND|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Sent ApiVersionRequest (v3, 52 bytes @ 0, CorrId 3) %7|1608650466.377|SSLVERIFY|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Broker SSL certificate verified %7|1608650466.377|CONNECTED|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Connected (#1) %7|1608650466.378|FEATURE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Updated enabled protocol features +ApiVersion to ApiVersion %7|1608650466.379|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Broker changed state CONNECT -> APIVERSION_QUERY %7|1608650466.379|SEND|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Sent ApiVersionRequest (v3, 52 bytes @ 0, CorrId 1) %7|1608650466.471|RECV|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Received ApiVersionResponse (v3, 358 bytes, CorrId 1, rtt 235.13ms) %7|1608650466.471|FEATURE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq %7|1608650466.472|STATE|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Broker changed state APIVERSION_QUERY -> UP %7|1608650466.473|SEND|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Sent MetadataRequest (v4, 26 bytes @ 0, CorrId 2) %7|1608650466.582|RECV|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Received ApiVersionResponse (v3, 358 bytes, CorrId 3, rtt 252.82ms) %7|1608650466.582|STATE|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state APIVERSION_QUERY -> UP %7|1608650466.584|SEND|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Sent MetadataRequest (v4, 26 bytes @ 0, CorrId 4) %7|1608650466.587|CONNECT|rdkafka#producer-3| [thrd:main]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Selected for cluster connection: no cluster connection (broker has 1 connection attempt(s)) %7|1608650466.588|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Received CONNECT op %7|1608650466.590|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state INIT -> TRY_CONNECT %7|1608650466.590|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: broker in state TRY_CONNECT connecting %7|1608650466.591|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state TRY_CONNECT -> CONNECT %7|1608650466.591|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Connecting to ipv4#222.222.222.222:443 (ssl) with socket 2128 %7|1608650466.635|RECV|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Received ApiVersionResponse (v3, 358 bytes, CorrId 1, rtt 255.64ms) %7|1608650466.635|FEATURE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD,SaslAuthReq %7|1608650466.636|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Broker changed state APIVERSION_QUERY -> UP %7|1608650466.637|SEND|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Sent MetadataRequest (v4, 26 bytes @ 0, CorrId 2) %7|1608650466.708|RECV|rdkafka#producer-2| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Received MetadataResponse (v4, 265 bytes, CorrId 2, rtt 235.52ms) %7|1608650466.822|RECV|rdkafka#producer-1| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Received MetadataResponse (v4, 265 bytes, CorrId 4, rtt 238.79ms) %7|1608650466.850|CONNECT|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Connected to ipv4#222.222.222.222:443 %7|1608650466.883|RECV|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-2-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/2: Received MetadataResponse (v4, 265 bytes, CorrId 2, rtt 245.91ms) %7|1608650467.333|SSLVERIFY|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker SSL certificate verified %7|1608650467.333|CONNECTED|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Connected (#2) %7|1608650467.334|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state CONNECT -> APIVERSION_QUERY %7|1608650467.335|SEND|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Sent ApiVersionRequest (v3, 52 bytes @ 0, CorrId 3) %7|1608650467.580|RECV|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Received ApiVersionResponse (v3, 358 bytes, CorrId 3, rtt 244.39ms) %7|1608650467.580|STATE|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Broker changed state APIVERSION_QUERY -> UP %7|1608650467.581|SEND|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Sent MetadataRequest (v4, 26 bytes @ 0, CorrId 4) %7|1608650467.824|RECV|rdkafka#producer-3| [thrd:ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxx]: ssl://kafka-prod-kafka-0-dev.apps.okdcluster.xxxxxxxxxxxxxxix.com:443/0: Received MetadataResponse (v4, 265 bytes, CorrId 4, rtt 242.22ms)

Serg675 commented 3 years ago

Hello,

Is periodic disconnection/connection mentioned above normal? Is it sign of some issue?

confluentinc / confluent-kafka-dotnet

External Access to Apache Kafka on OpenShift/OKD from the .NET client. Is it possible? #1485

Description

Checklist