morganstanley / modern-cpp-kafka

A C++ API for Kafka clients (i.e. KafkaProducer, KafkaConsumer, AdminClient)
Apache License 2.0
331 stars 86 forks source link

Cannot connect to kafka with SSL protocol #219

Closed persona94 closed 10 months ago

persona94 commented 10 months ago

I'm trying to connect to kafka using ssl. I used the conan package for modern-cpp-kafka. I seem to be getting a property error when I try to set the security protocol to ssl

`

Properties producerProperties({ { "bootstrap.servers", {brokersList } } });

producerProperties.put("log_cb", [](int level, const char* /*filename*/, int /*lineno*/, const char* msg)
    {
            std::cout <<  msg;
      });

producerProperties.put("error_cb", [](const kafka::Error& error)
    {
       std::cout << error.message();
    });

if (useSSL)
{
    producerProperties.put("security.protocol", "ssl")
        .put("ssl.ca.location", options.caFile)
        .put("ssl.certificate.location", options.certLocation)
        .put("ssl.key.location", options.keyFile);
}
producer = std::make_unique<Producer>(producerProperties);

I get an error in the log

KafkaProducer[219dd444-cbc95e50] failed to be initialized with property[security.protocol:ssl], result[-1]
KafkaProducer[219dd444-cbc95e50] failed to be initialized with property[ssl.ca.location:/kafka/ca.crt], result[-1]
KafkaProducer[219dd444-cbc95e50] failed to be initialized with property[ssl.certificate.location:/kafka/user.crt], result[-1]
KafkaProducer[219dd444-cbc95e50] failed to be initialized with property[ssl.key.location:/kafka/user.key], result[-1]
KafkaProducer[219dd444-cbc95e50] initializes with properties[bootstrap.servers=kafka-kafka-bootstrap.kafka:9093|client.id=219dd444-cbc95e50|log_level=5]

From the driver code it appears that the error happens in rd_kafka_conf_set when called from KafkaClient::KafkaClient

const rd_kafka_conf_res_t result = rd_kafka_conf_set(rk_conf.get(), k.c_str(), v->c_str(), errInfo.str(), errInfo.capacity()); if (result == RD_KAFKA_CONF_OK) { _properties.put(prop.first, prop.second); } else { KAFKA_API_DO_LOG(Log::Level::Err, "failed to be initialized with property[%s:%s], result[%d]", k.c_str(), v->c_str(), result); }

But the errorInfo doesn't get sent to the callback function.

I built the conan packages with

conan install -if build . --settings compiler.version=11 -o librdkafka:ssl=True

I'm using modern-cpp-kafka/2023.03.07 which seems to be the latest version available thru' Conan.

persona94 commented 10 months ago

The issue was a stale docker cache not building with ssl options enabled.