outbox-pattern-in-go/

[utterances-bot]

Implementing the Outbox pattern in go | pkritiotis Software Engineering

This article presents the design and implementation of the outbox pattern in go.

https://pkritiotis.io/outbox-pattern-in-go/

[iangregsondev]

Great write-up! I have one concern. What about the frequency of the outbox polling/checking. Imagine that I am currently executing something, and I require that the message is pushed onto the queue immediately - of course - if the queue is available.

Is there a way to say "Send now - otherwise store in the outbox" has these been thought about or implemented in the outbox implementation?

So if I can send it immediately, this would be the preferred option. Otherwise, it would go to an outbox and get processed and retried up to the threshold. If it ends up in the outbox, I have no option but to wait for it to be processed when the queue becomes available.

Thanks for a great write-up!

[pkritiotis]

Hey @iangregsondev! 👋 Thank you for the comment! Valid concern! The current implementation introduces a delay between the outbox sending time and the message delivery time, as you correctly point out.

Unfortunately, we cannot avoid storing the message in the outbox table and therefore we cannot completely avoid a delay, but we could minimize it.

The benefit of the outbox pattern lies in executing multiple operations as a unit of work in one atomic transaction which then guarantees the "at least once delivery". So "either everything or nothing is stored in the database". For this reason, we have a strict rule when implementing the outbox pattern; We must store the message in the database before attempting to deliver the message in the queue.

Now, can we do something to minimize the polling frequency delay?

Yes we can! 💪

Once we commit the transaction, we can follow the below steps using a targeted processing method

1. Lock the message that we have just stored successfully in the outbox table
2. Attempt to deliver the message to the message broker
3. Store the result
4. Unlock the message

If the outbox pattern is implemented in a custom fashion without using a generic-purpose outbox library, all good. We just need to call the above code after we commit the transaction 🎉

If we want to provide this functionality in a general-purpose library, I can see the following options:

1. Our generic outbox package/library can expose a targeted message "instant delivery" method that follows the above steps. And this should be executed by the consumer of the library in cases that the consumer needs to minimize the delay.
2. Another (quite inflexible) option would be to enforce the commit of the SQL transaction within the publisher.Send method. Once the method is called, the SQL transaction is executed, and if successful, we attempt to deliver the message following the steps mentioned above. This could be another method of the publisher that could be used in a case-by-case scenario.

Hope this helps! 🙏