r3wind
commented
2 years ago Using K6 I was able to emulate the negotiate endpoint flow and web socket connection to Azure SignalR service, publishing hundreds of thousands of messages in a period of minutes without any mitigation available. Not only does this put significant load through to our Blazor Applications but it is huge risk in terms of Denial of Wallet.
We already have Azure Front Door in place for which the initial negotiate request goes through, this does give us some level of protection for DDoS and allowing an individual to open up thousands of connections. However, as @Jonno12345 alluded to, once a connection is made there is nothing stopping that individual flooding the websocket connection indefinitely.
Were there any plans for this feature to be implemented in future? This is currently a huge blocker for us right now to be able to deploy Blazor Server applications that are supported by Azure SignalR service. It is a huge flaw which I'm surprised doesn't have more of a spotlight?
@Jonno12345 Did you find any other mitigations for this issue?
All of these messages were sent from a single K6 client, not even scratching the surface of what could potentially be sent:
1,394,441 messages received and 1,575,046 sent in 10 minutes.
Jonno12345
dbContext
Is your feature request related to a problem? Please describe.
We are migrating from a ASP NET Signalr with Redis backplane service, to a dedicated ASP NET Core Signalr + Azure Signalr backend service. In this process, we have discovered how vulnerable we will be, as we cannot find a way to ensure a single user doesn't swamp all available connections on our units. As part of our testing to ensure the system works as intended, we used an adaptation of Crankier and found that, within minutes, a single person could take all 5000 connections on our example 5 unit Azure SignalR service.
In an attempt to mitigate this, we investigated using something like Azure Front Door, only to realise this doesn't support websockets and doesn't give us enough rate limit configuration. Our next step was to try and hide the service behind haproxy, allowing us to limit the maximum number of concurrent connections. However, to do this we would need to route through another domain, and we receive 401 errors when the server attempts to make a connection to the Azure SignalR service, due to the JWT being sent having a mismatched bearer token, where the audience is now set to our haproxy domain instead of *.service.signalr.net.
We are able to protect our ASP.Net Core server to prevent a user from hitting our negotiate method too many times, however once they have a connection we have no way to handle this, and there's nothing stopping the user still taking every available connection on our units, just over a longer period of time.
Describe the solution you'd like
Initially I considered being able to override the audience value in the JWT, which could be done if we had public access to the interface IServiceEndpointGenerator https://github.com/Azure/azure-signalr/blob/937a0fb58dc4bade9086aa1fb9d0eae9fab77804/src/Microsoft.Azure.SignalR/EndpointProvider/IServiceEndpointGenerator.cs#L10 and this was passed through via DI, however this would still reveal a public endpoint in the JWT to potential attackers.
We require a way to secure this service, assuming there isn't already one. A way to limit connections per client IP address would be a useful feature, being able to route successfully through a rate-limiting Azure service, or a way to tell the Azure SignalR server to expect a different audience (eg. custom domain support) to be able to also add layer 4 protection (assuming this isn't already factored in).
Otherwise, any suggestion you have for how we can prevent a malicious user from attacking our Azure SignalR service, be that a regular DDoS or a way to prevent the user from using a methodology similar to Crankier (or even simply opening thousands of browser tabs) and taking all available connections would be appreciated.