search

liftbridge-io / go-liftbridge

Go client for Liftbridge. https://github.com/liftbridge-io/liftbridge
Apache License 2.0
66 stars 18 forks source link

durable load balanced stream? #112

Closed justinrush closed 3 years ago

justinrush commented 3 years ago

One use case I have for liftbridge is a scale out pub/sub model I can use to load balance across a backend (similar to typical rabbitMQ use cases). The documentation for liftbridge and the SDK documentation seem to indicate that having a load balanced group listening on the same stream is possible, but I can't figure out how that is done. The closest I've seen is the SetCursor option, but that doesn't appear to be implemented in the client returned by Connect.

It looks like there is something called an ActivityStream that was recently introduced into liftbridge, but not available in the go API. Is that something that accomplishes what I'm looking for and I just need to patiently wait?

tylertreat commented 3 years ago

A load-balance group is a set of streams that receive messages from a NATS subject in round-robin fashion. This is done by creating multiple streams with the same subject and using the Group option with the same name. You could use this pattern to balance messages amongst a set of consumers.

justinrush commented 3 years ago

A load-balance group is a set of streams that receive messages from a NATS subject in round-robin fashion. This is done by creating multiple streams with the same subject and using the Group option with the same name. You could use this pattern to balance messages amongst a set of consumers.

But the Publish API requires that you send to a stream name and not a nats subject?

Provided I'm reading the wrong thing and I need to call something else, how does the consumer checkpoint so Liftbridge knows that the message was successfully delivered?

tylertreat commented 3 years ago

Yeah, that is correct. It's more meant for the NATS layer, i.e. publishing with NATS clients to NATS subjects, rather than the Liftbridge stream layer. This probably needs to be documented better.

If you are working only at the streams layer, the better approach might be to use a single stream that is partitioned, then have a single consumer for each partition which checkpoints its position. This pattern will become a lot more seamless once consumer groups are available.

justinrush commented 3 years ago

OK, i'll check back in a few months then, thanks