search

confluentinc / librdkafka

The Apache Kafka C/C++ library
Other
264 stars 3.15k forks source link

[Feature request] Vectored I/O for producer #3176

Open viccpp opened 3 years ago

viccpp commented 3 years ago

Please consider adding something like writev() to producer functions.

In my application I have messages that consist of segments (this allows me to use fast pool allocator). Before calling produce() I have to allocate a new buffer for each message and linearize it there. This almost kills all benefits of using memory pool (one-segment messages don't need to be linearized).

viccpp commented 3 years ago

Similar - #2219

edenhill commented 2 years ago

How large is each segment?

viccpp commented 2 years ago

256 bytes

edenhill commented 2 years ago

I've done some performance benchmarks with io vectoring, and it turns out that it is actually cheaper to copy smaller chunks into a larger single memory area than it is to pass around an iovec array, which then also limits the number of messages that can be sent in a single sendmsg() call due to IOV_MAX.

Alexander-Shukaev commented 2 years ago

Which is 1024 these days on most platforms.

edenhill commented 2 years ago

When using iovecs for the payload we also need to pass at least one extra iovec per message for the message headers.