Any plans to support NATS?

[natiki]

Hi,

Was just wondering if there were any plans to support https://nats.io/?

[mookid8000]

Yes! 😁 is it something you would be interested in using?

[natiki]

@mookid8000 yes I plan to use it alongside Kubeless and Kong.

[mookid8000]

Have you used Nats for anything before?

[natiki]

No

[natiki]

FWIW I will be starting to use the streaming version of the server https://nats.io/documentation/streaming/nats-streaming-intro/

[mookid8000]

Cool 🙂 I'll be happy to hear about your experiences, especially a judgment on how Nats streaming might/might not fit into the Rebus model.

[artyom-p]

Also, still interested in it. https://github.com/rebus-org/Rebus/issues/707

[mookid8000]

@xxsybreedxx funny, I just read a little bit about Nats yesterday, because I was watching a video about Aeron.... and then I came across Nats Streaming.

A thing that struck me yesterday about Nats, was that it's not durable – at has "at most once delivery", which means that Rebus will NOT readily be able to use it as a transport.

I think Nats Streaming would be a better fit, although its Kafkaesque model would require some careful work to make fit into Rebus.

@natiki Did you get started using Nats Streaming?

[natiki]

@mookid8000 still doing some early stuff.....

In terms of persistence I came across https://www.infoq.com/news/2018/08/nats-liftbridge / https://github.com/liftbridge-io/liftbridge. Still need to look at that in detail but ultimately it brings persistence to NATS and a "Kafkaesque" experience to NATS. I still need to investigate this further and LiftBridge is still Alpha.

[mookid8000]

I'll close this issue now, as I don't think NATS would be such a good Rebus transport.

As I mentioned, NATS Streaming supports more of the properties that Rebus likes/prefers/requires (especially durability), BUT NATS Streaming is more like Kafka than a traditional queue... therefore, I think NATS Streaming would work better as a broker implementation in Topos (which is kind of like Rebus, but for log-based message brokers, like Kafka and Azure Event Hubs – and is VERY immature right now 😁 )

[natiki]

@mookid8000 Did you look at the vanilla NATS which does not support durability? NATS Streaming is built on top of NATS to add that. I think you will find vanilla NATS to be closer to what you expect.

As for durability.... As Rebus supports RabbitMQ which includes durability why is durability an exclusion factor?

[mookid8000]

(...) I think you will find vanilla NATS to be closer to what you expect.

Did you mean the opposite here? If not, then I think not – vanilla NATS is more like TCP, so there's no durable "store-and-forward" middleware involved.

NATS Streaming, however, can provide message persistence, so that would be more well-suited. But as I mentioned, I think Rebus does not lend itself very well to the streaming nature of event brokers like NATS Streaming and Kafka – I think Topos is a better fit for that.

[djradon]

This issue has been closed a long time, but I think NATS JetStream support would be great. I'm so curious why Rebus does not lend itself to the streaming nature of some event brokers.

[mookid8000]

I'm so curious why Rebus does not lend itself to the streaming nature of some event brokers.

Messaging with streams is just different from messaging with queues.

Queues generally have the property that a message can be consumed by one recipient only, and then it gets ACKed (i.e. deleted from the queue). Event streams generally have the property that they can be read, sequentially, by any number of clients, all of whom simply keep track of how far they have come.

When consuming messages from queues, you generally can't make too many assumptions about the ORDER of messages, ESPECIALLY if a message could potentially be a re-delivered message that has been sitting in a dead-letter queue for a while. OTOH when working with log-based brokers, you CAN make assumptions about ordering, as messages withing a given partition of a specific topic will be ordered.

The properties mentioned so far probably is the explanation why messages from queues tend to model isolated items of work, whereas messages in streams generally tend to be connected to their "neighbors" (i.e. must be consumed in the context of the stream they're in).

The APIs in Rebus are designed to be good at working with queue-based messaging, so when your message handler is invoked, there's no built-in way to know if the message is a part of a stream or to figure out the context. In fact, if processing your message takes just a tiny bit of time (or especially if it happens to await something), Rebus will most likely be processing its neighboring message concurrently. This makes the order of processing non-deterministic, which can be a great scaling trick if your messages can be handled in parallel (which they should) - but if you were working with a stream of messages, you could never have parallelism greater than the number of partitions.

Queues tend to be quick to connect to - log-based brokers tend to have much more protocol associated with producing/consuming messages, because it might need to distribute partitions within topic between a number of consumers withing a consumer group.

I know this wasn't a very well-structured reply 😓 but hopefully you can get an impression why there's a pretty big impedance mismatch between the queue-based and the log-based messaging worlds.

[djradon]

That's an excellent reply, thank you!

FWIW, "With JetStream a stream can also be used as a queue by setting the retention policy to WorkQueuePolicy and leveraging pull consumers to get easy horizontal scalability of the processing."

[mookid8000]

I don't know how JetStream works and what it adds on top of plain NATS, and I can't leave out the possibility that it could be done and might also work just fine. I'll be happy to guide you if you want to experiment with building a Rebus transport that uses JetStream to do its thing 🙂