Serverless: Extremely unreliable broadcast until restart after burst of simultaneous broadcasts (10+/s)

[stephenvisser]

Background:

• Azure SignalR Service is configured Serverless
• ASP.NET Core servers (Azure App Service) use the Management API to initiate broadcast messages
• We have both regular and on-demand messages. We saw this last week and reduced a 'ping' type broadcast message from every second to every 10 seconds. We just experienced the issue again despite this change.
• In some cases, we are sending 10+ broadcasts per second, perhaps sustaining to 40 over 5 seconds
• We sporadically see failure rates of perhaps > 50% when broadcasting.
• All connected clients tend to experience a high failure rate of messages. There are no obvious characteristics (longevity of connection, recency of connection, browser/machine type) that make it more or less likely to miss a message. While one client may experience an issue receiving a particular message at one instant, it receives the same message the next minute.
• It's not clear whether non-broadcast messages are affected; they are less frequent so it's difficult to observe.
• This behaviour does not appear to be rectified just by leaving it alone (there are spans of many seconds where we'd expect queues have a chance to clear out). Only by restarting the service do we see reasonable reliability again

We are aware of the guidance here that states broadcast messages via the REST API should not exceed a rate of 2/s.

Our average message rate over the course of a minute will typically be well below 1/s. The strange thing for us is that even if we have a burst of 10 or more messages in a given second, this by itself is not sufficient to create the issue. It can be days or weeks before we see this 'limp mode'.

[ccic]

If you guess there are stability issue, could you please run this benchmark on serverless broadcast?

dotnet tool install --global Microsoft.dotnet-signalr-benchmark --version 1.0.0-preview2.19521.1

using the following YAML:

mode: simple
config:
  connectionString: Endpoint=https://xxx;AccessKey=xxx
  connections: 100
  baseSending: 1
  step: 1
  sendingSteps: 1
scenario:
  name: restBroadcast

More details about the benchmark tool can be found: https://github.com/Azure/azure-signalr-bench