search

Azure / azure-functions-kafka-extension

Kafka extension for Azure Functions
MIT License
114 stars 78 forks source link

need control in setting the commit offset value (i.e manual commit) with Kafka extension instead of auto commit #431

Open thilakbe opened 1 year ago

thilakbe commented 1 year ago

At high load, we see duplicate events consumption issue when we use Function app with Kafka trigger. Hence, we are looking for a feature where we could set the offset commit value manually instead of auto commit which is managed by Azure functions platform.

PSanetra commented 1 year ago

We need to set commit offset manually for a different reason as we have a use case where we need to cache events for a specific time and flush the events if a duration or memory threshold is reached. We want to commit after flushing and not when the events where received.

marcelmedina commented 2 weeks ago

+1

This should be a standard capability available with Azure Functions.

I've seen many integrations where teams decided to leverage their own worker services due to the lack of this capability in Azure Functions.