search

confluentinc / librdkafka

The Apache Kafka C/C++ library
Other
7.36k stars 3.11k forks source link

Fails to set security.protocol to SSL #4699

Closed pksilen closed 2 months ago

pksilen commented 2 months ago

Description

LibRdKafka fails to set the security.protocol to SSL.

It reports the following error to logs:

Broker name "SSL://tls-kafka:9099" parse error: protocol "SSL" does not match security.protocol setting "plaintext"

As shown below (how to reproduce section), I have set the global config prop "security.protocol" correctly to "ssl", but it seems not to be set according to above error message claiming "security.protocol" is "plaintext" still.

According to these intructions: https://github.com/confluentinc/librdkafka/wiki/Using-SSL-with-librdkafka

I have verified that WITH_SSL is set to 1 in config.h. Only difference is that I have installed openssl-devel package instead of installing openssl and libssl-dev, because the latter ones are for Debian/Ubuntu and openssl-devel is what is found in Rocky Linux. Could this be the root cause of the problem? Have you tested with Rocky/RHEL Linux?

How to reproduce

Set the global config before creating a producer:

if (m_globalConf->set("bootstrap.servers",  "SSL://tls-kafka:9099", errorMsg) != RdKafka::Conf::CONF_OK)
        {
        return false;
        }

if (m_globalConf->set("security.protocol", "ssl", errorMsg) != RdKafka::Conf::CONF_OK)
        {
            return false;
        }

        if (m_globalConf->set("ssl.cipher.suites", m_kafkaProperties.cipherList, errorMsg) != RdKafka::Conf::CONF_OK)
        {
            return false;
        }

        if (m_globalConf->set("ssl.key.location", m_kafkaProperties.keyFile, errorMsg) != RdKafka::Conf::CONF_OK)
        {
            return false;
        }

        if (m_globalConf->set("ssl.certificate.location", m_kafkaProperties.certFile, errorMsg) != RdKafka::Conf::CONF_OK)
        {
            return false;
        }

        if (m_globalConf->set("ssl.ca.location", m_kafkaProperties.caFile, errorMsg) != RdKafka::Conf::CONF_OK)
        {
            return false;
        }

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

pksilen commented 2 months ago

Here are debug level logs from the librdkafka with the following debug list: generic,broker,topic,security,protocol

<135>1 2024-04-26T06:35:20.412Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaTopicControl BROKER [thrd:app]: Broker name "SSL://tls-kafka:9099" parse error: protocol "SSL" does not match security.protocol setting "plaintext"
<134>1 2024-04-26T06:35:20.412Z exportica-streaming Exportica - LOG [hr@28458 level="INFO"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaTopicControl ERROR [thrd:app]: No brokers configured
<135>1 2024-04-26T06:35:20.412Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaTopicControl CONFWARN [thrd:app]: No `bootstrap.servers` configured: client will not be able to connect to Kafka cluster
<132>1 2024-04-26T06:35:25.413Z exportica-streaming Exportica - LOG [hr@28458 level="WARNING"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar Failed to create topic: Failed to create topic: Broker: Request timed out
<135>1 2024-04-26T06:35:25.413Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar OPENSSL [thrd:app]: Using OpenSSL version OpenSSL 1.1.1k  FIPS 25 Mar 2021 (0x101010bf, librdkafka built with 0x101010bf)
<135>1 2024-04-26T06:35:25.413Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar SSL [thrd:app]: Setting cipher list: AES128-SHA:AES256-SHA:DHE-RSA-AES256-SHA:AES128-SHA256:AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDH-RSA-AES128-SHA:ECDH-RSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:ECDH-RSA-AES128-SHA256:ECDH-RSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDH-RSA-AES128-GCM-SHA256:ECDH-RSA-AES256-GCM-SHA384
<135>1 2024-04-26T06:35:25.415Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar SSL [thrd:app]: Loading CA certificate(s) from file /tmp/config/exportica.ca.pem
<135>1 2024-04-26T06:35:25.418Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar SSL [thrd:app]: Loading public key from file /tmp/config/exportica.cert.pem
<135>1 2024-04-26T06:35:25.419Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BRKMAIN [thrd::0/internal]: :0/internal: Enter main broker thread
<135>1 2024-04-26T06:35:25.419Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BROKER [thrd:app]: ssl://tls-kafka:9099/bootstrap: Added new broker with NodeId -1
<135>1 2024-04-26T06:35:25.419Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar CONNECT [thrd:app]: ssl://tls-kafka:9099/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
<135>1 2024-04-26T06:35:25.419Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar INIT [thrd:app]: librdkafka v2.3.0 (0x20300ff) rdkafka#producer-2 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,sasl_oauthbearer, CC CXX PKGCONFIG INSTALL GNULD LDS C11THREADS LIBDL PLUGINS ZLIB SSL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x287)
<135>1 2024-04-26T06:35:25.419Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar DESTROY [thrd:app]: Terminating instance (destroy flags none (0x0))
<135>1 2024-04-26T06:35:25.419Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:app]: Interrupting timers
<135>1 2024-04-26T06:35:25.419Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:app]: Sending TERMINATE to internal main thread
<135>1 2024-04-26T06:35:25.419Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:app]: Joining internal main thread
<135>1 2024-04-26T06:35:25.419Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:main]: Internal main thread terminating
<135>1 2024-04-26T06:35:25.420Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar DESTROY [thrd:main]: Destroy internal
<135>1 2024-04-26T06:35:25.420Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BROADCAST [thrd:main]: Broadcasting state change
<135>1 2024-04-26T06:35:25.420Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar DESTROY [thrd:main]: Removing all topics
<135>1 2024-04-26T06:35:25.420Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar DESTROY [thrd:main]: Sending TERMINATE to ssl://tls-kafka:9099/bootstrap
<135>1 2024-04-26T06:35:25.420Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:main]: Purging reply queue
<135>1 2024-04-26T06:35:25.420Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:main]: Decommissioning internal broker
<135>1 2024-04-26T06:35:25.420Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:main]: Join 2 broker thread(s)
<135>1 2024-04-26T06:35:25.420Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERM [thrd::0/internal]: :0/internal: Received TERMINATE op in state INIT: 2 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
<135>1 2024-04-26T06:35:25.420Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar FAIL [thrd::0/internal]: :0/internal: Client is terminating (after 0ms in state INIT) (_DESTROY)
<135>1 2024-04-26T06:35:25.420Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar STATE [thrd::0/internal]: :0/internal: Broker changed state INIT -> DOWN
<135>1 2024-04-26T06:35:25.420Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BROADCAST [thrd::0/internal]: Broadcasting state change
<135>1 2024-04-26T06:35:25.420Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BRKTERM [thrd::0/internal]: :0/internal: terminating: broker still has 2 refcnt(s), 0 buffer(s), 0 partition(s)
<135>1 2024-04-26T06:35:25.420Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd::0/internal]: :0/internal: Handle is terminating in state DOWN: 1 refcnts (0x2248c80), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
<135>1 2024-04-26T06:35:25.420Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar FAIL [thrd::0/internal]: :0/internal: Broker handle is terminating (after 0ms in state DOWN) (_DESTROY)
<135>1 2024-04-26T06:35:25.419Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BRKMAIN [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Enter main broker thread
<135>1 2024-04-26T06:35:25.421Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar CONNECT [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Received CONNECT op
<135>1 2024-04-26T06:35:25.421Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar STATE [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Broker changed state INIT -> TRY_CONNECT
<135>1 2024-04-26T06:35:25.422Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BROADCAST [thrd:ssl://tls-kafka:9099/bootstrap]: Broadcasting state change
<135>1 2024-04-26T06:35:25.423Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Handle is terminating in state TRY_CONNECT: 2 refcnts (0x224f260), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
<135>1 2024-04-26T06:35:25.423Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERM [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Received TERMINATE op in state TRY_CONNECT: 2 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
<135>1 2024-04-26T06:35:25.424Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar FAIL [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Client is terminating (after 1ms in state TRY_CONNECT) (_DESTROY)
<135>1 2024-04-26T06:35:25.424Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar STATE [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Broker changed state TRY_CONNECT -> DOWN
<135>1 2024-04-26T06:35:25.425Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BROADCAST [thrd:ssl://tls-kafka:9099/bootstrap]: Broadcasting state change
<135>1 2024-04-26T06:35:25.425Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BRKTERM [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: terminating: broker still has 2 refcnt(s), 0 buffer(s), 0 partition(s)
<135>1 2024-04-26T06:35:25.426Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar CONNECT [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: broker in state DOWN connecting
<135>1 2024-04-26T06:35:25.426Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar STATE [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Broker changed state DOWN -> CONNECT
<135>1 2024-04-26T06:35:25.427Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BROADCAST [thrd:ssl://tls-kafka:9099/bootstrap]: Broadcasting state change
<135>1 2024-04-26T06:35:25.431Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar CONNECT [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Connecting to ipv4#172.20.0.6:9099 (ssl) with socket 8
<135>1 2024-04-26T06:35:25.433Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Handle is terminating in state CONNECT: 1 refcnts (0x224f260), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
<135>1 2024-04-26T06:35:25.433Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar FAIL [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Broker handle is terminating (after 6ms in state CONNECT) (_DESTROY)
<135>1 2024-04-26T06:35:25.433Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar STATE [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Broker changed state CONNECT -> DOWN
<135>1 2024-04-26T06:35:25.433Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BROADCAST [thrd:ssl://tls-kafka:9099/bootstrap]: Broadcasting state change
<135>1 2024-04-26T06:35:25.434Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:main]: Internal main thread termination done
<135>1 2024-04-26T06:35:25.434Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:app]: Destroying op queues
<135>1 2024-04-26T06:35:25.434Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:app]: Destroying SSL CTX
<135>1 2024-04-26T06:35:25.434Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:app]: Termination done: freeing resources
<135>1 2024-04-26T06:35:27.516Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaTopicControl BROKER [thrd:app]: Broker name "SSL://tls-kafka:9099" parse error: protocol "SSL" does not match security.protocol setting "plaintext"
<134>1 2024-04-26T06:35:27.516Z exportica-streaming Exportica - LOG [hr@28458 level="INFO"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaTopicControl ERROR [thrd:app]: No brokers configured
<135>1 2024-04-26T06:35:27.516Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaTopicControl CONFWARN [thrd:app]: No `bootstrap.servers` configured: client will not be able to connect to Kafka cluster
<132>1 2024-04-26T06:35:32.517Z exportica-streaming Exportica - LOG [hr@28458 level="WARNING"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar Failed to create topic: Failed to create topic: Broker: Request timed out
<135>1 2024-04-26T06:35:32.517Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar OPENSSL [thrd:app]: Using OpenSSL version OpenSSL 1.1.1k  FIPS 25 Mar 2021 (0x101010bf, librdkafka built with 0x101010bf)
<135>1 2024-04-26T06:35:32.517Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar SSL [thrd:app]: Setting cipher list: AES128-SHA:AES256-SHA:DHE-RSA-AES256-SHA:AES128-SHA256:AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDH-RSA-AES128-SHA:ECDH-RSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:ECDH-RSA-AES128-SHA256:ECDH-RSA-AES256-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDH-RSA-AES128-GCM-SHA256:ECDH-RSA-AES256-GCM-SHA384
<135>1 2024-04-26T06:35:32.521Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar SSL [thrd:app]: Loading CA certificate(s) from file /tmp/config/exportica.ca.pem
<135>1 2024-04-26T06:35:32.523Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar SSL [thrd:app]: Loading public key from file /tmp/config/exportica.cert.pem
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BRKMAIN [thrd::0/internal]: :0/internal: Enter main broker thread
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BROKER [thrd:app]: ssl://tls-kafka:9099/bootstrap: Added new broker with NodeId -1
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar CONNECT [thrd:app]: ssl://tls-kafka:9099/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar INIT [thrd:app]: librdkafka v2.3.0 (0x20300ff) rdkafka#producer-4 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_plain,sasl_scram,plugins,sasl_oauthbearer, CC CXX PKGCONFIG INSTALL GNULD LDS C11THREADS LIBDL PLUGINS ZLIB SSL HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x287)
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar DESTROY [thrd:app]: Terminating instance (destroy flags none (0x0))
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:app]: Interrupting timers
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:app]: Sending TERMINATE to internal main thread
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BRKMAIN [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Enter main broker thread
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:app]: Joining internal main thread
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar CONNECT [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Received CONNECT op
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar STATE [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Broker changed state INIT -> TRY_CONNECT
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BROADCAST [thrd:ssl://tls-kafka:9099/bootstrap]: Broadcasting state change
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Handle is terminating in state TRY_CONNECT: 3 refcnts (0x23c4e00), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:main]: Internal main thread terminating
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar DESTROY [thrd:main]: Destroy internal
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BROADCAST [thrd:main]: Broadcasting state change
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar DESTROY [thrd:main]: Removing all topics
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar DESTROY [thrd:main]: Sending TERMINATE to ssl://tls-kafka:9099/bootstrap
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:main]: Purging reply queue
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:main]: Decommissioning internal broker
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERM [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Received TERMINATE op in state TRY_CONNECT: 2 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar FAIL [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Client is terminating (after 0ms in state TRY_CONNECT) (_DESTROY)
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar STATE [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Broker changed state TRY_CONNECT -> DOWN
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BROADCAST [thrd:ssl://tls-kafka:9099/bootstrap]: Broadcasting state change
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BRKTERM [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: terminating: broker still has 2 refcnt(s), 0 buffer(s), 0 partition(s)
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:main]: Join 2 broker thread(s)
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar CONNECT [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: broker in state DOWN connecting
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar STATE [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Broker changed state DOWN -> CONNECT
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BROADCAST [thrd:ssl://tls-kafka:9099/bootstrap]: Broadcasting state change
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERM [thrd::0/internal]: :0/internal: Received TERMINATE op in state INIT: 2 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar FAIL [thrd::0/internal]: :0/internal: Client is terminating (after 0ms in state INIT) (_DESTROY)
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar STATE [thrd::0/internal]: :0/internal: Broker changed state INIT -> DOWN
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BROADCAST [thrd::0/internal]: Broadcasting state change
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BRKTERM [thrd::0/internal]: :0/internal: terminating: broker still has 2 refcnt(s), 0 buffer(s), 0 partition(s)
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd::0/internal]: :0/internal: Handle is terminating in state DOWN: 1 refcnts (0x22bf170), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
<135>1 2024-04-26T06:35:32.524Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar FAIL [thrd::0/internal]: :0/internal: Broker handle is terminating (after 0ms in state DOWN) (_DESTROY)
<135>1 2024-04-26T06:35:32.525Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar CONNECT [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Connecting to ipv4#172.20.0.6:9099 (ssl) with socket 9
<135>1 2024-04-26T06:35:32.525Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Handle is terminating in state CONNECT: 1 refcnts (0x23c4e00), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
<135>1 2024-04-26T06:35:32.525Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar FAIL [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Broker handle is terminating (after 0ms in state CONNECT) (_DESTROY)
<135>1 2024-04-26T06:35:32.525Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar STATE [thrd:ssl://tls-kafka:9099/bootstrap]: ssl://tls-kafka:9099/bootstrap: Broker changed state CONNECT -> DOWN
<135>1 2024-04-26T06:35:32.525Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar BROADCAST [thrd:ssl://tls-kafka:9099/bootstrap]: Broadcasting state change
<135>1 2024-04-26T06:35:32.525Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:main]: Internal main thread termination done
<135>1 2024-04-26T06:35:32.525Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:app]: Destroying op queues
<135>1 2024-04-26T06:35:32.525Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:app]: Destroying SSL CTX
<135>1 2024-04-26T06:35:32.525Z exportica-streaming Exportica - LOG [hr@28458 level="DEBUG"] [origin enterpriseId="28458" software="exportica-streaming" swVersion="3.3.3"] RdKafkaProducer exportica.amf_pulsar TERMINATE [thrd:app]: Termination done: freeing resources
pksilen commented 2 months ago

This was a logic issue in our code