search

twmb / franz-go

franz-go contains a feature complete, pure Go library for interacting with Kafka from 0.8.0 through 3.7+. Producing, consuming, transacting, administrating, etc.
BSD 3-Clause "New" or "Revised" License
1.78k stars 182 forks source link

How to fetch zero records without waiting #752

Closed muxmuse closed 4 months ago

muxmuse commented 4 months ago

If no new records are available on the cluster, Client.PollRecords (or PollFetches) waits for the first records to arrive.

Is it possible to check the cluster for new records and immediately return if no records were found instead of waiting?

twmb commented 4 months ago

There is not a way to trigger the client to send a request, check the cluster, and then (this is what's not supported) confirm that the fetch response returned with no records.

One thing you can do is to use a nil context to PollFetches, which triggers the function to return immediately. This does not answer if any records are available on the broker, but it does at least answer if any records are available that the client has fetched.

If you want to answer what's available on the broker, better is to use kadm.ListEndOffsets and compare the returned offsets to the last offset you fetched -- this is an exact answer, and somewhat the purpose of that function.