search

trytouca / trytouca

Continuous Regression Testing for Engineering Teams
https://touca.io/docs
Apache License 2.0
489 stars 24 forks source link

Server-sent events may not work correctly if multiple server instances are running #411

Open Rossh87 opened 1 year ago

Rossh87 commented 1 year ago

Description

Delivery of server-sent events to clients will become unreliable if more than 1 instance of touca-app is running as a cluster/single service. This may not be an important configuration to support, but I thought I'd flag it. Feel free to close if not relevant.

Environment

Any configuration that can deploy multiple instances of the server container, e.g. K8s, Docker Swarm.

Steps To Reproduce

Deploy as per Environment above, with at least 2 instances of touca-app running. Connect multiple clients to touca-app. Complete an action from one of the clients that should trigger the delivery of an event to all connected clients. The event may or may not be broadcast to all appropriate clients.

Expected Behavior

All clients should receive all relevant events, regardless of which instance of touca-app they are connected to.

Additional Context

Not a high-priority fix, but it will definitely come up if you ever need to horizontally scale Touca.

It happens because BullMQ does not support a 'fanout' job distribution pattern, and it doesn't seem like they will any time soon. Since each instance of touca-app keeps track of server-event subscriptions in its own process, and each enqueued job is consumed by exactly one worker and then discarded, only the clients who happen to be connected to the same process as the worker that consumes the job can receive the event.

ghorbanzade commented 1 year ago

Hi @Rossh87. Great point. Thanks for raising this issue. This is an important limitation to keep in mind, even if supporting this setup configuration (horizontal scaling) is not a high priority at the moment. Do you have any suggestions on how to overcome this limitation?

Rossh87 commented 1 year ago

If you ever need a quick fix for this specific issue, I think redis streams can do what you need within the current architecture.

IMO it's more likely to come up in the context of a general move towards distributed services. You may find you want to move some of the cron processes out into their own service, have separate machines doing batch processing, or that you need better failover behavior. At that point, this issue will be one consideration among many, and you may realize you no longer want Redis in the mix at all. So if you can, I would wait until you understand those other considerations better.