[QUERY] Inquiry on EventHubsProducerClient SendIdempotentAsync

[minghuaw]

Library name and version

Azure.Messaging.EventHubs

Query/Question

It seems to me that Service Bus and Event Hubs take different approaches towards duplication mitigation. SB performs this on the service side with uuid generated on the client side, while EH tries to achieve this with SendIdempotentAsync.

I vaguely remember somewhere it says the EH was built on top of SB (I am very likely wrong on this). But I was just wondering why these two services take two different approaches?

In addition, I am a bit confused by SendIdempotentAsync, because this seems like something that should be already taken care of by the AMQP 1.0 protocol (the section where it discusses link recovery). Part of my confusion comes from how I am trying to mimic the dotnet sdk in rust. Because of rust's ownership model, the underlying AMQP sender takes the ownership of the message and also takes care of resolving duplication upon recovery. And thus I feel like implementing something similar to SendIdempotent is not necessary.

Environment

No response

[jsquire]

Service Bus performs de-duplication based on the MessageId on the service-side within a window of time. It's also worth mentioning that de-duplication does not depend on using the same AMQP link, so the service has responsibility for tracking the messages that it has seen within that window. This is a heavy-weight and fairly expensive process that does not fit the "Event Hubs is optimized to maximize throughput" positioning.

Event Hubs also lacks the concept of an identity for an event. While MessageId is available, it intentionally has no meaning to the broker and is intended to be part of the implicit contract between producers and consumers. As Event Hubs as built in support for the Kafka protocol, it has been necessary to also add support for some of Kafka's features that were not supported by Event Hubs. The idempotent publishing model is one of those. Because this idempotent publishing approach had to exist for use with the Kafka protocol, it was decided to also expose it via AMQP.

Idempotent publishing is a shallow, best-effort approach that aims to reduce duplication, not eliminate it. It has plenty of gaps, sharp edges, and requires a non-trivial amount of state tracking in the client. It also forces the client and service to maintain common state - but without a way for them to coordinate and ensure that there's no drift.

Kafka's model and approach is described here. Some of the work-in-progress design materials for the Event Hubs client implementation can be found below. It's important to note that these are developer notes and not official documentation - expect them to be point-in-time and potentially not fully accurate against the current state of things.

• The approach we used when moving from beta to GA. This highlights many of the concerns that we had after several months and multiple user studies, detailing the reasons why we restricted this to the EventHubBufferedProducerClient.

• Initial design thoughts for conversations

• A distillation of the rules and corner cases, after some discussion

• Those distilled rules with usage examples for illustration

[jsquire]

That all said, my personal opinion is that it's not worth including in the Rust SDK. We have seen virtually no interest from customers for this feature since it was released. In the year that it was in beta and being tested with focus groups, it was neither well understood nor met with any enthusiasm once the limitations were known. (Some customers were really excited for Service Bus-style duplicate detection, and came away disapointed.)

There is one internal team using this in the raw form with the non-buffered producer due to special needs. In the 1.5 years since they launched with it, there has been a non-trivial amount of discussions needed between them, the Event Hubs service team, and the Azure SDK team to get the implementation correct and stable. The complexity and rough spots are not worth the efforts in the vast majority of scenarios.

AMQP sender takes the ownership of the message and also takes care of resolving duplication upon recovery.

This is true for recovering a link within a client instance. However, it doesn't handle recovering after a host crash/migration when a new client must be created. (the service does not support AMQP's durable terminus; there's no way to recover a previous link state)

[github-actions[bot]]

Hi @minghuaw. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.