outbox-pattern-implementation-challenges/

[utterances-bot]

Outbox pattern - Why, How and Implementation Challenges | pkritiotis Software Engineering

This article presents the outbox pattern, a communication/messaging pattern used in event-driven distributed systems that allows executing database operations and reliably publishing messages.

https://pkritiotis.io/outbox-pattern-implementation-challenges/

[mariokom]

Great article Pan! Could we use optimistic concurrency to eliminate the concern "when an instance fails to unlock the message"? Wondering what will be the ratio of failed updates which will come as a result of the optimistic concurrency and if some kind of randomization will mitigate that.

[pkritiotis]

@mariokom thanks for your comment :smile: :pray:. Very interesting point, you are right it could work!

Still everything is a tradeoff; Optimistic concurrency would avoid the locking/unlocking management complexity but at the same time it would probably be less efficient. Since the purpose of locking/unlocking requirement in this case is to block multiple instances from delivering a message externally (rather than updating the data record state), we could end up with multiple dispatcher instances delivering the same message; which can't be rolled back. Randomization would help in 'uniquely' choosing messages. Depending on the use-cae the message rate with conflict ratio would be a useful factor to assist in this decision.

In any case, consumer idempotency is there to guard against this scenario, and outbox by nature guarantees the 'At least one delivery' so it is definitely a valid option.

Great idea, thanks for pointing this out :rocket:

[amasiero]

Amazing explanation Pan! The article is so easy to understand all the steps and application. I have two questions, one technical and another just a curiosity.

I was thinking about the complexity of the payload in the outbox table. Should we considered to use a NoSQL database?

Second question is about the diagrams software. What do you use? They are really great.

Thanks in advance.

[pkritiotis]

Hey @amasiero , I'm very glad you found this article helpful 😄 🙏

I was thinking about the complexity of the payload in the outbox table. Should we considered to use a NoSQL database?

One of the constraints of the outbox table updates is that they have to be committed within an atomic transaction which includes other entities/tables inserts or updates. This means that you need a data store that offers transactions across tables. Most NoSQL databases don't support transactions across multiple data items, so you cannot guarantee the atomicity, which is mandatory.

Second question is about the diagrams software. What do you use? They are really great.

Thank you for the compliment; for my diagrams, I use the awesome drawio

[amasiero]

@pkritiotis, thanks for the explanation and tip ☺️