Inquiry About Message Pre-fetching in Confluent Kafka Go Client

[dhruwill]

Description

I am currently working with the Confluent Kafka Go client and wanted to inquire about the possibility of implementing a feature where messages can be queued (pre-fetched \ keeps writing to a buffered channel) while the consumer is busy processing previous messages. I understand application itself can implement such a feature, but wanted to check if Is there an already existing mechanism within the client to achieve this?

How to reproduce

Checklist

Please provide the following information:

• [ ] confluent-kafka-go and librdkafka version (LibraryVersion()):
• [ ] Apache Kafka broker version:
• [ ] Client configuration: ConfigMap{...}
• [ ] Operating system:
• [ ] Provide client logs (with "debug": ".." as necessary)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[milindl]

There is a prefetch functionality in the Go client, however, internally, in the C library which is backing this client, we prefetch the messages. They're kept within the C code, however, and not a channel. However, if all you want to do network requests in parallel to processing messages in your consumer, that's already happening.

For any other use, you would need to write the buffering on the application level.

[dhruwill]

Hi @milindl, can you please point me to the documentation/code for the prefetch functionality in the Go client?
Also, is the prefetch functionality in C client configurable ? i.e If I want to prefetch a certain number of messages, can i configure it ? Any documentation on this, where i can read on how many messages it keeps prefetched etc.. will help Thank you!

[milindl]

It's configurable, see https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md

The properties you might start by looking at might be queued.max.messages.kbytes. You can look at the other properties around it too, they're all somewhat related to fetching (like how frequently we'll try to fetch etc.)