search

harlow / kinesis-consumer

Golang library for consuming Kinesis stream data
MIT License
264 stars 90 forks source link

Interval Batching for Separate Callback Function #128

Open epociask opened 3 years ago

epociask commented 3 years ago

It would be amazing to have a separate functionality with an auxiliary callback function that instead takes a slice of records as its parameter to allow for processing of a batch records upon some time elapsed interval. It could be passed through an additional scan function called something like ScanBatch. It would invoke the reading of records from kinesis and pass them through a callback function every-time the time interval is triggered, allowing for batch reading and providing the ability for many records to be consumed at once. :)

harlow commented 3 years ago

Hi thanks for the note. Agree that could be a nice interface. I've just used a channel in the callback func in the past, but agree this would be a little cleaner

epociask commented 3 years ago

Thank you so much for the response, I opened up a PR for this. If you could please review it and let me know what particular needs changing that'd be incredible. Looking forward to help contribute to this awesome library :)