batch publish

[vtolstov]

Please add BatchPublish method to supports slice of messages, in this case we don't drop support for old code and allows to publish batch of messages via single request to nats server this minimize syscall overhead in case of sending many small messages (i have iso8583 json messages from card transaction engine)

[derekcollison]

Please try async publish.

    toSend := 1_000_000
    start := time.Now()
    for i := 0; i < toSend; i++ {
        js.PublishAsync("foo", []byte("OK"))
    }
    <-js.PublishAsyncComplete()
    tt := time.Since(start)
    fmt.Printf("time is %v\n", tt)
    fmt.Printf("%.0f msgs/sec\n", float64(toSend)/tt.Seconds())

[vtolstov]

In this i need to use adync error handler that receives via callback single message. And bug drawback i need to. Have all messages publushed in order. In async case does it guarantee?

[derekcollison]

It guarantees the order. Each message still sends pubAck etc like normal they are just processed out of band and async.

[chrispoulsen]

I'm currently investigating NATS / JS trying to figure out whether it could be used as an event store in an event sourcing system. It feels like some kind of batch publish would be a good building block for that.

It would allow for cases like a microservice receiving a command message, sourcing and applying it to an aggregate that in turn produce one or more domain events. The domain events should ideally be published atomically (all or none) and then the command message could finally be ack/nack'ed depending on the publish result.

NATS is new to me, so if there are better ways to handle this case then I'm all ears.

Edit: Just noticed this seems to be in the go client project, we're building on the JVM - but if it is possible to do here, then I guess it can be done in the Java client as well

[derekcollison]

We do not have the ability to batch or lightweight TX, but we do allow conditional publishes based on sequence, msg header etc.

Take a look at the docs and join the Slack channel which has a JetStream sub channel.

[chrispoulsen]

Thank you for the swift reply! I have a hard time coming up with a good solution without being able to ensure the atomic part on mapping from one command message to multiple domain events. My imagination currently stretches to the classic approach of using a SQL server to ensure consistency (transactional outbox/transaction log tailing etc. to maintain enough event info for NATS to be able to de-dup in case a service needs to recover from a crash during publishing), but that feels a lot like trying to use the wrong tool for the job - If those things needs to be in place anyway, there is not much left that really sets NATS apart for this use case.

[derekcollison]

With what we already have you could do this IMO.

Receive command message. Generate and send domain events with msg_id that is unique but reprodicable (for dedupe etc). Ack command message.

So if you fail anywhere you will get the command message back and will try again, any domain events that succeeded will be de-duped and will be ok. Any ones that did not make it will make it.

[chrispoulsen]

Thanks again! Then I'll give it a shot with some PoC code as I really like the other value propositions of the project. There could be some fiddling around sourcing up the aggregate (from events) that the command has to work on, in case of a crash - But I think there are a couple of ways to handle that cleverly like event publish order or the reproducible ids you mention.

[danyo1399]

With what we already have you could do this IMO.

Receive command message. Generate and send domain events with msg_id that is unique but reprodicable (for dedupe etc). Ack command message.

So if you fail anywhere you will get the command message back and will try again, any domain events that succeeded will be de-duped and will be ok. Any ones that did not make it will make it.

That doesn’t really work as it should be all or nothing. Publishing one message at a time could allow another process to raise events between first and last message of a batch invalidating the rest of the batch messages leaving the system in an inconsistent state.