search

infinyon / fluvio

Lean and mean distributed stream processing system written in rust and web assembly. Alternative to Kafka + Flink in one.
https://www.fluvio.io/
Apache License 2.0
3.79k stars 500 forks source link

Offset management #4089

Closed jayeshpk1 closed 1 month ago

jayeshpk1 commented 3 months ago

I would like to retrieve and start processing from the last offset commit. Is there an API to retrieve the latest offset for a topic/consumerid/partition combination in a distributed environment?

digikata commented 3 months ago

Hi @jayeshpk1,

You can start processing from the latest offset from the API by requesting an Offset::end() or often you want Offset::from_end(1) in the consumer config.

Or you can save tracking offsets for a given consumer id. The documentation for that kind of offset management is here:

https://www.fluvio.io/docs/fluvio/concepts/produce-consume#consumer-offsets

Feel free to ask any follow ups or share your use case in case if you have any more questions

ajhunyady commented 1 month ago

Closing, as no additional action needed.