dashbitco / broadway_kafka

A Broadway connector for Kafka
233 stars 53 forks source link

Add support for reseting offsets to a specific timestamp #136

Closed danmarcab closed 8 months ago

danmarcab commented 8 months ago

👋

We have a use case where we need to consume from a topic, but neither consuming from the start or the end is acceptable. We'd like to start consuming, let's say, the last day worth of messages.

One way this could work is allowing {:timestamp, timestamp} to be a value for the offset_reset_policy option. timestamp could be a timestamp. :brod can use that timestamp to resolve the offsets. This timestamp would be fixed so maybe even better if we could pass a function that returns a timestamp.

I have a working branch and will open a draft PR so the amount of changes needed can be evaluated, but I am mainly looking to see if this is something that would be accepted.

As always, thanks for the amazing library 🙇

slashmili commented 8 months ago

Seems like a good addition specially if it's already in brod!