search

Azure / azure-event-hubs-node

Node client library for Azure Event Hubs https://azure.microsoft.com/services/event-hubs
MIT License
50 stars 46 forks source link

Option to not checkpoint the offset during lease-renewal #97

Closed davidwmartines closed 6 years ago

davidwmartines commented 6 years ago

Is your feature request related to a problem? Please describe. Current offset gets saved during lease renewal, which happens outside scope of message handling, even if message handling code does not call context.checkpoint(). This could happen if messageHandler takes some time to process and eventually gets an error, in which case we don't want to checkpoint the current offset, so that we can fix the issue, restart the EPH and try to process that event again.

Describe the solution you'd like Some option on EPH to prevent the offset from being saved as part of the lease renewal, so the burden of checkpointing is strictly on the user. This way I can be sure current offset is only committed when I call context.checkpoint() after all processing of the event has completed successfully.

amarzavery commented 6 years ago

That makes sense. Currently the EPH checkpoints on behalf of the customer every time a lease is renewed. I shall make it customizable.

amarzavery commented 6 years ago

There is a tradeoff between checkpointing and balancing EPH instances across partitions. If checkpointing is delayed, then when another instance of EPH (say eph2) starts, it will receive checkpoint info that is relatively older than what the current EPH instance(say eph1) is currently receiving. Thus eph2 will receive some messages that were already received by eph1. Thus you would need additional metadata in your application that would indicate that the message was already processed or the effect of processing a message should be idempotent.

amarzavery commented 6 years ago

azure-event-processor-host@0.1.4 has been published where you can disable auto checkpointing by setting autoCheckpoint: false as one of the options while creating the EPH.

davidwmartines commented 6 years ago

Confirmed autoCheckpoint: false working as requested. Thanks!