Jetstream: transactional batch publish messages

[sloveridge]

Feature Request

New pub style analogous to fetch for pull based consumers to allow multiple messages for a stream to be published with a single call and all succeed or fail (effectively a transaction). Messages can be for multiple subjects in the same stream.

Use Case:

There are cases where multiple messages must all successfully publish to ensure consistent state on the system. An example of this using ORDERS would be when an incoming order will be fulfilled by multiple locations and has its line items split on publish. A single order may then result in multiple messages with a format such as:

ORDERS.{locationId}.{itemId}

If there is a connection failure in the middle of publishing the orders messages the systems state is now inconsistent (a partial order is in the stream). The ideal way of handling this would be to publish all of these messages in a single batch which would succeed / fail as a whole. Note messages should not be available to consumers until the batch was successful.

Another scenario where this may be useful is to allow clients to buffer messages before sending, ie bulk insert.

Proposed Change:

Using the Go client as an example something along the lines of:

type batch struct {
    subj String
    payload [][]byte
}

PublishBatch(batches ...batch, opts ...PubOpt) (*PubAck, error)

Who Benefits From The Change(s)?

• Applications requiring transaction based publishing of events
• Applications with high publish rates of small messages

Alternative Approaches

• An application could attempt to do a rollback in the situation of connection failure mid publish (ie delete events for the partial orders line items) however as connection was lost these may be published to the stream a significant time afterwards. This is problematic if there are downstream systems immediately acting on the initial line item pub. Additionally in the case that the upstream application dies mid publish it may be difficult to determine that the order was only partially published. Effectively the rollback can be handled via distributed transactions however these can be quite complex to implement successfully and it is much simpler to catch it on the initial publish.
• An application could publish all the line items as a single "batch" message however this removes the ability to direct traffic to consumers based on subject, removing a core benefit of Jetstream in this scenario.

[suikast42]

You cant retry 50-55 and get them in order again, you cant stop and start again at 50 cos 56+ got saved.

So the 56 will be detect as duplicate will not be redilvered to the client, right?

As I see see with async send there is no chanche to guranty the full ordering. With sync push I got no this trouble but have less throughput right ?

[ripienaar]

Yes, thats my point, you cant be sure that ordering is correct when doing batches and retrying some. If you need strict ordering you cant do batches.

[ramonberrutti]

I have been working on this, and one solution that I found is to send the messages with a specific header and store them in memory until the last message contains a commit message.

Headers:

// Headers for published messages.
const (
    JSTransactionId       = "Nats-Transaction-Id"
    JSTransactionSequence = "Nats-Transaction-Sequence"
    JSTransactionCommit   = "Nats-Transaction-Commit"
)

type stream struct {
        // ...
    // transaction
    tx map[string]*txState
        // ...
}

type txState struct {
    msgs []*StoreMsg
    lseq uint64
}

The hard work is to refactor processJetStreamMsg to validate multiple msg simultaneously and each StoreMsgs function for memory and filesystem store.

More TODOs are to roll back and expire the transaction.

[sysradium]

That would be a great feature.

For example let's take an aggregate which reacts to a certain command. This results in a few domain events being created.

I pop them out of the aggregate and, using optimistic locking to make sure that no-one else had modified my aggregate, push them to nats. But what if 3 out of 5 messages succeed and the 4th fails? Ok, I reload my aggregate to reapply message 4 and 5 but I realize that a certain invariant can no longer be satisfied so I can't proceed.

Now I have an aggregate potentially in a weird state which is valid but incomplete. If we don't have a batch publisher what might a workaround?