Configuration for web application (load balanced) with multiple subscribers

[DouglasHammon-FV]

Hello. Great job with Rebus, awesome product.

Can you please recommend how to configure Rebus for the following:

• MS SQL Server for transport and subscriptions
• Web application in a Kubernetes cluster that will have 0..N number of instances of the web app running behind a load balancer. The number of instance is dynamic and can be auto scaled. So each instances needs to subscribe itself.
• Be able to publish a message and have each web app instance subscribe and handle the same message

I'm would like to avoid introducing another piece of infrastructure (besides MS SQL Server), so trying to avoid something like Redis Pub/Sub.

I currently have Rebus configured, but when a message is published, only one instance of the web app handles the message, instead of each instance.

Thanks for the help!

[mookid8000]

Web application in a Kubernetes cluster that will have 0..N number of instances of the web app running behind a load balancer. The number of instance is dynamic and can be auto scaled. So each instances needs to subscribe itself.

The problem with this is that you have two options: Either

1. share an input queue (i.e. competing consumers), or
2. have uniquely named input queues for each instance

Solution (1) will, as you've already discovered, not work for e.g. invalidating caches (or whatever it is that you need to do) in a clustered environment, because only one of the instances will receive each event. This is a good thing, because this is what enables scaling out workers and is a super-easy way of distributing load.

Solution (2) could work for you, but it would have the slightly wonky problem that it would leave queues (and therefore also messages!) behind when scaling down. If the queue names were random on each boot-up, the left-behind queue would simply sit there and keep receiving events forever until you manually removed the queue and its subscriptions (in the opposite order, obviously). If the queue names were stable (i.e. pod n would use queue n) then it would start up and receive all the events received while being away. Neither strategy would be ideal.

So.... as you can see, Rebus might not really be what you're after here. If you tell me more about the problem you are trying to solve then maybe I can help you in the right direction.

[DouglasHammon-FV]

@mookid8000 Thanks for taking the time and replying. You points make sense and are valid.

The problem I'm trying to solve is: My application needs to support sending updates (SSE and Long Polling) to clients (other applications) for when data is modified and the clients should invalidate their cache and get new values from my application.

Solution 2 would work IF I could reliably remove the subscription when the application was shutdown (or have a cleanup task). To your point, not an ideal solution.

For now, I think what I'm going to do is have each instance of my app poll the database for changes in a table and cause that to trigger the application to send those updates to downstream clients. If/when the application grows beyond this, I think I'll need to use something like Redis Pub/Sub. It supports more of a dynamic subscribe flow.

Thanks again for your reply.