Closed arielmoraes closed 5 years ago
Hi @arielmoraes, that's a very good question! 👍
And the fact is that you're right, there's no global atomicity between microservices and that's actually on purpose!
Doing so would require the use of distributed transactions, using two phase commits, but that would cause coupling between the microservices, which is one of the key issues that we want to avoid in a microservices architecture, where each microservice has to be as independent as possible.
So, in a microservices architecture we are actually looking for eventual consistency by using an asynchronous message-based communication system, to propagate changes between microservices, in a pub/sub pattern.
And, just as you realized, doing so in a resilient way is very hard and requires more complexity than the one handled in eShopOnContainers right now.
Some ways to achieve the required resiliency for a production application would be using:
eShopOnContainers is just a sample to showcase architectural patterns and is not intended to be used in production as is, so the implementation of the Pub/Sub pattern is a basic one based on RabbitMQ.
You can get more details about this issue in @CESARDELATORRE's Microservices Architecture eBook on sections:
Hope this helps.
Actually this is happening inside just one microservice, the Ordering microservice, which has two databases, one for storing the integration events and one for storing the aggregates. So the atomicity here is not between microservices but inside the Ordering microservice. I think the main idea is to save the aggregate only if the integration event is persisted, is that correct?
In this page of the book there are two sections explaining the part related to the domain events:
So the problem is that there are two atomic operations happening here one is for the OrderingContext and the related domain events and one is for the OrderingContext and the IntegrationEventContext, but in fact all of them should be inside the same atomic operation, shouldn't them?
Hi @arielmoraes,
It looks like I didn't get you right to begin with, will try to get it better this time.
Based on your input I traced calls, commands and events and seemed to get to a point where a DB commit is performed while dispatching a domain event cascading from another domain event, before the processing of first event finishes (the chained domain events you mentioned). But it got pretty complex at some point so will have to review it.
Anyway, it looks like atomicity is compromised here, but further investigation is necessary.
As for your previous comment, just a couple of clarifications:
There's only one Ordering database that holds the tables for both the aggregates and the integration events logging.
The strategy for handling domain events and integration events in eShopOnContainers should be (according to the documentation):
Commit all changes in the microservice and insert the integration event log (atomically) as ready to publish in the one transaction and then
Publish the integration event and mark the event as published in another transaction.
As the same document states, this strategy was chosen for the sake of simplicity, although it's known it doesn't handle all possible failure modes. (Take a look at page 139 from https://aka.ms/microservicesebook for a more recent content)
So, besides the "small" issue with atomicity you pointed out, it was a design decision to have two transactions, and leaving a trace for diagnosing. Although this is not probably adequate for a production application.
Another point is that there seems to be a issue with the implementation.
Hope this is a bit clearer now, and thanks a lot for your insights.
Marking this as a bug, there seems to be an important issue with transaction atomicity, it will be reviewed in depth.
This issue wasn't supposed to be closed, wonder who did it 🙄
Hi @arielmoraes,
This was the chosen solution was, at high level:
Begin a transaction in the TransactionBehaviour: https://github.com/dotnet-architecture/eShopOnContainers/blob/e05a87658128106fef4e628ccb830bc89325d9da/src/Services/Ordering/Ordering.API/Application/Behaviors/TransactionBehaviour.cs#L40
While processing the related commands, add the resulting integration events to a list (an outbox): https://github.com/dotnet-architecture/eShopOnContainers/blob/e05a87658128106fef4e628ccb830bc89325d9da/src/Services/Ordering/Ordering.API/Application/Commands/CreateOrderCommandHandler.cs#L38 There's a nice detail here, and it's that the outbox is persisted in the DB, so it'd be easy to implement a "watchdog" microservice that would handle possible failures in the publishing mechanism. This is not implemented in eShopOnContainers.
Commit the transaction for all changes when returning from the behavior pipeline: https://github.com/dotnet-architecture/eShopOnContainers/blob/e05a87658128106fef4e628ccb830bc89325d9da/src/Services/Ordering/Ordering.API/Application/Behaviors/TransactionBehaviour.cs#L44
Publish all integration events in the outbox: https://github.com/dotnet-architecture/eShopOnContainers/blob/e05a87658128106fef4e628ccb830bc89325d9da/src/Services/Ordering/Ordering.API/Application/Behaviors/TransactionBehaviour.cs#L48
This strategy also handles the atomicity of all changes mentioned in issue https://github.com/dotnet-architecture/eShopOnContainers/issues/721
So I'm closing this issue now, but feel free to comment, will reopen if needed.
In the Ordering service we have a
ValidateOrAddBuyerAggregateWhenOrderStartedDomainEventHandler
which handles theOrderStartedDomainEvent
, after the Buyer is added or updated a call toSaveEntitiesAsync
is made to dispatch the domain events and save the changes to the database, after that, a call toPublishThroughEventBusAsync
is made. So, if the changes was already saved by calling theSaveEntitiesAsync
method no atomicity will be in place or am I missing something? There is even a bigger problem if a chain of domain events are raised because only the last call toPublishThroughEventBusAsync
will have atomicity between the current handled aggregate and the event.