New MatsImpl: "Consume - Process - Produce in single DB transaction, with async move of messages, w/dedup"

[stolsvik]

Need an acronym, or a name, for such a style..!

As suggested by and discussed with @tsoiland and @ksaua, the idea is effectively as such: The StageProcessor will read messages from an incoming message DB table, and write to an outgoing DB table. The main point is that these tables will reside in the same DB as in which the stage-lambda executes, so that the DB transaction will literally span the entire operation-set consume-process-produce, and they will literally all be done, or none be done.

So, how does the incoming messages appear, and how are the outgoing messages being passed on?

There are several ways to accomplish this - but the logic is that there is two (or more) separate processing making sure this happens.

• Receive: Messages are received in one way or the other - and will be deduplicated upon insert, using some GU messaging id added by the sender. The receiver will send back to the sender which messages it has actually persisted, so that the sender can "cross them out" on its side. Once a new message is persisted, a StageProcessor can be pinged.

• Send: Another thread waits for new messages to appear on the outgoing table (and can be pinged by the StageProcessor when it has written one down). Once it appears, it will send it to the correct recipient for storage in its incoming table. It will however not mark it as "sent" (or delete it) before it has gotten acknowledge from the receiver that it has actually been stored. It can redeliver messages multiple times, as the receiver will notice if it gets a double delivery and just inform the sender about this, "I've already gotten that message, and also these since you probably wonder..".

• StageProcessors: Notice that the messages will be marked as consumed, not deleted. This is to enable catching of double deliveries after it has processed them. A reaper can optionally come and cull the table for messages older than X days - or put them into a different table if it is interesting to keep a record of all messages having passed through.

• Non-persistent messaging: It is easy to not store the messages: Instead just keep them in memory on both the receive and send side. Notice that this gives some interesting aspects in that senders should then load-balance between the instances it sends to, as the messages then aren't "load balanced" automatically by residing on common storage backend (the database), but instead just residing in memory of the node that received the message.

One interesting (obvious?) candidate for communicating between nodes is ZeroMQ, and the Java-variant JeroMQ. I am not certain how the nodes would find each other, tho.

[stolsvik]

Another project to check out wrt. messaging is Aeron: https://github.com/real-logic/aeron

[stolsvik]

Wrt. acknowledging messages - and multiple nodes per endpoint:

• The sender would again have to acknowledge that the receiver acknowledged a received message before the receiver can delete them from its incoming table. When the sender gets the acknowledge, it can delete the message from its outgoing queue as it sends the ack-ack. If the sender gets the receive-acknowledge once more (due to the communication having dropped the first receive-ack), it would find that the message was not present in its outgoing queue anymore, and thus ack-ack it right away.
• The message 1. appears in the outgoing queue of the sender, 2. appears in the incoming queue at receiver, 3. is deleted from the outgoing queue of the sender, 4. is deleted from the incoming queue of the receiver (after it is both processed and ack-ack'ed).
• There is some interesting elements wrt. multiple nodes for one endpoint: It all boils down to each of them using the same backend database. Thus, it does not matter if the receive-acknowledgement (or ack-ack mentioned above) is handled by the node that actually sent the message, or another node.

[stolsvik]

The "outbox pattern" (#77) is pretty much half of this solution.