search

centiservice / mats3

Mats3: Message-based Asynchronous Transactional Staged Stateless Services
https://mats3.io/
Other
60 stars 5 forks source link

Make a simple way to perform "the outbox pattern". #23

Open stolsvik opened 5 years ago

stolsvik commented 5 years ago

The outbox pattern is a way to overcome the rather bad scenario in messaging: You receive a message, start a DB transaction, perform some state changing operations on the DB, send an outgoing message, commit the DB - and when about to commit the MQ to both acknowledge the processing of the incoming message, and push the outgoing message onto the MQ, the MQ connection goes down, the MQ crashes, or the micro service crashes or whatever other nasty stuff affecting your commit of the MQ transaction.

The point is that you have now performed the state changing operation on the database, but you have neither acknowledged the receipt of the incoming message, nor had the outgoing message pushed to the MQ.

You will now most probably get a redelivery of the incoming message. But it should stop processing right there, as you have double-delivery catching in place. But that effectively means that the process has stopped.

The "outbox pattern" - which evidently have gotten lots of attention in 2019 - basically sticks the outgoing message on the DB as an intermediate step. The point is that it is then committed together with the DB-commit of the other state changing messages. Another process picks up the outgoing messages from the outbox and pushes them "onto the wire". This process can also crash, but it should crash in a way that ensures that the message will be delivered at least once. And since all messaging endpoints should either be idempotent, or have double-delivery catching, that won't be a problem.

One random link from Google: http://www.kamilgrzybek.com/design/the-outbox-pattern/

stolsvik commented 4 years ago

Make sure to expose this on the public API, so that I can use it in MatsSockets. That is, when initiating a message, instead of doing the actual send, request or publish, ask for a "outbox representation", returning a byte array that can be stored. This must be possible then to "post to outbox" (with or without backing db).

Could be a method "returnOutbox", which then gave the byte array as a part of the returned value from the send, request and publish methods.