search

reiseburo / hermann

A gem providing cross-platform Kafka producer and consumer support
MIT License
71 stars 77 forks source link

Can we expose offsets up into the Consumer? #83

Closed rtyler closed 9 years ago

rtyler commented 9 years ago

Thinking aloud, I can imagine certain usecases where the consumer might want to hold onto their current offset, partition tuple for logging, etc.

rtyler commented 9 years ago

So I can answer this now, here's the answer for posterity!

On JRuby, since we're relying on the "high level consumer" API from Kafka's client libraries, we cannot get access to the offset being consumed. It is not exposed. it's possible to get the offset when implementing a "simple" (read: basic) consumer but then the user is responsible for implementing their own leader following and error handling code. I'm exploring this work in my beetle experimenting, which might prove useful at some point in the future here.

For MRI, I believe we can get at the offsets, but I do not believe it is worth introducing without API symetery between JRuby and MRI