[Question] How to make sure my event was processed Exactly-Once in my load-balance group

[hung-doan]

Say, I'm using Azure Service Bus for Event Bus.

My application is deployed with 05 instanced, which was placed behind a load-balancer.

In my Basket Service, I publish "UserCheckoutAcceptedIntegrationEvent" to event bus and I expected that my Ordering Service will process "UserCheckoutAcceptedIntegrationEvent" event only once.

But there are 05 Ordering Services are subscribing for "UserCheckoutAcceptedIntegrationEvent".

How can I guarantee that "UserCheckoutAcceptedIntegrationEvent" is processed by exactly one Service in my load-balancer group. 4 others Ordering Service instance will ignore this event until It was processed.

I know that Azure Service Bus support Exactly-One processing feature (it is mentioned here )

But, I don't want my event to be processed once globally, I just want It's processed by "Ordering Service" once. Because my Service has many instances for load-balancing

[mvelosop]

Hi @hung-doan, that's a good point that's handled by the UserCheckoutAcceptedIntegrationEventHandler.

Each UserCheckoutAcceptedIntegrationEvent has an Id (Guid) that's saved in the ordering.requests table when an order is created, so the IdentifiedCommand<CreateOrderCommand, bool>, can check if there's an order already created with that Id, to just ignore the message so just one order is created.

The ordering database then becomes the single point of control that ensures only one order is created.

Hope that makes the scenario clear.

[CESARDELATORRE]

@mvelosop I'm not fully sure about that. The IdentifiedCommand<CreateOrderCommand, bool> ensures that the same original message is IDEMPOTENT, so the same message is processed just once, in case it was sent several times through the network because of retries or any other reason.

But not sure if that is also ensuring that it won't be processed from multiple instances of the same target microservice. Could be, but I'm still not sure. We'll check out how the pattern is implemented in the code and will come back to this thread, soon.

@hung-doan About what you say 'I don't want my event to be processed once globally, I just want it to be processed by the "Ordering Service"' --> that has to be ensured/controlled by the application.

In any case, we'll come back to this thread as soon as we review how the pattern was implemented, ok?

Thanks for the feedback! 👍

[mvelosop]

Hi @CESARDELATORRE, I got to trace the code to the point where the event idempotency is based on checking the uniqueness of the RequestId in the ordering.requests table.

It's pretty clear in the unit test at IdentifiedCommandHandlerTest.Handler_sends_no_command_when_order_already_exists.

[CESARDELATORRE]

@mvelosop - Probably. I'm also now reviewing the code and even when the Integration Events at the message broker (RabbitMQ or Azure Service Bus) are processed once per subscription (which is right because they are Events and any integration with any service or app might be interested on that event), the fact that these particular event is processed just once for the same order is controlled by the application based on the ID in the database added with this line of code at the UserCheckoutAcceptedIntegrationEventHandler:
var requestCreateOrder = new IdentifiedCommand<CreateOrderCommand, bool>(createOrderCommand, eventMsg.RequestId);

The point is that the UserCheckoutAcceptedIntegrationEvent is an integration Event, and therefore, Events can be processed multiple times by multiple subscriptions.

But its UserCheckoutAcceptedIntegrationEventHandler triggers a business Command (a command should be executed/processed just once) and that is controlled by the original message ID. I think this mechanism we implemented ensures IDEMPOTENCY for messages that are sent several times because of networking retries but also ensures that the same order is not processed multiple times by multiple subscriptions.

I believe that when deploying to AKS/Kubernetes we tested that scenario (multiple-instances of the Order microservice), but just in case, we'll test it again when possible.

We'll double-check reviewing further the code and we'll come back to this thread. Do not close the issue until we confirm 100%, ok?

[sm-g]

Is there a mistake with saving changes? first ClientRequest saved https://github.com/dotnet-architecture/eShopOnContainers/blob/a9676b4b5667eeebe0aa8ff5f64c604e9bd3971a/src/Services/Ordering/Ordering.Infrastructure/Idempotency/RequestManager.cs#L40

then everything else https://github.com/dotnet-architecture/eShopOnContainers/blob/a9676b4b5667eeebe0aa8ff5f64c604e9bd3971a/src/Services/Ordering/Ordering.API/Application/Commands/CancelOrderCommandHandler.cs#L35

I did not find that these saves are in same transaction.

[mvelosop]

Hi @sm-g, I think you've got a point here!

If there's any problem between inserting in the request table and committing the command (e.g. CreateOrder) then the order will never be created (or whatever) because the command had been registered as processed.

That's the problem right?

[sm-g]

Yes, problem with extra Save

[mvelosop]

So, i'm marking this a potential bug, for further review, because we would need some way to create that condition in a failing test.

Then it would also need to handle the situation where the commit fails with a duplicate in the ordering.requests PK, in which case it would just have to be ignored, which is the default action when a command is tried twice.

BTW, You might want to summit a PR for the test and the solution!

And, hey, thanks for the heads up! :thumbsup:

[mvelosop]

Hi @hung-doan,

The issue we identified her was handled as part of a solution that covers issues https://github.com/dotnet-architecture/eShopOnContainers/issues/700, https://github.com/dotnet-architecture/eShopOnContainers/issues/721, and https://github.com/dotnet-architecture/eShopOnContainers/issues/828. The solution at high level, can be explained like this:

1. Begin a transaction in the TransactionBehaviour: https://github.com/dotnet-architecture/eShopOnContainers/blob/e05a87658128106fef4e628ccb830bc89325d9da/src/Services/Ordering/Ordering.API/Application/Behaviors/TransactionBehaviour.cs#L40

2. While processing the related commands, add the resulting integration events to a list (an outbox): https://github.com/dotnet-architecture/eShopOnContainers/blob/e05a87658128106fef4e628ccb830bc89325d9da/src/Services/Ordering/Ordering.API/Application/Commands/CreateOrderCommandHandler.cs#L38 There's a nice detail here, and it's that the outbox is persisted in the DB, so it'd be easy to implement a "watchdog" microservice that would handle possible failures in the publishing mechanism. This is not implemented in eShopOnContainers.

3. Commit the transaction for all changes when returning from the behavior pipeline: https://github.com/dotnet-architecture/eShopOnContainers/blob/e05a87658128106fef4e628ccb830bc89325d9da/src/Services/Ordering/Ordering.API/Application/Behaviors/TransactionBehaviour.cs#L44

4. Publish all integration events in the outbox: https://github.com/dotnet-architecture/eShopOnContainers/blob/e05a87658128106fef4e628ccb830bc89325d9da/src/Services/Ordering/Ordering.API/Application/Behaviors/TransactionBehaviour.cs#L48

So I'll close this issue now, but feel free to comment, will reopen if needed.