search

morganstanley / modern-cpp-kafka

A C++ API for Kafka clients (i.e. KafkaProducer, KafkaConsumer, AdminClient)
Apache License 2.0
331 stars 86 forks source link

is kafka:: clients:: producer:: KafkaProducer thread safe? #214

Closed sanjitk7 closed 11 months ago

sanjitk7 commented 12 months ago

I have an application that handles incoming data in an asynchronous thread via recurring callbacks - imagine a queue of data objects that are being processed and pushed to a Kafka endpoint. Each data object is processed inside a separate callback (all callbacks in the same thread).

I have a global producer object that is shared by all the callbacks for the 'Kafka Push'. Everything works fine with syncSend but the is see some form of data corruption (see garbage values) in the pushed data with the default (async) send. Is this normal?

My first thought was that shared global object is causing incorrect interleavings. But isn't the Kafka Producer thread safe by design?

sanjitk7 commented 11 months ago

Turns out I had to use send(Record, deliveryCb, option=KafkaProducer::SendOption::ToCopyRecordValue).