search

Azure / azure-signalr

Azure SignalR Service SDK for .NET
https://aka.ms/signalr-service
MIT License
427 stars 101 forks source link

Timeout when adding connection to group #1145

Open parbergius opened 3 years ago

parbergius commented 3 years ago

Description

At certain times, we get clusters of TimeoutExceptions in our logs when trying to add a connection to a group. The exception does not tell us exactly what´s wrong and it doesn´t seem possible to get any logs from Azure SignalR Service.

We have about 3000 groups and about 3 connections per group. Every second a new message is sent to each group. Could it be that the Azure SignalR Service is overloaded?

To Reproduce

public class RealtimeHub : Hub { public async Task Subscribe(string subjectKey) { string connectionId = Context.ConnectionId; await Groups.AddToGroupAsync(connectionId, subjectKey) <-- throws System.TimeoutException return 200;
} }

Exceptions

Ack-able message Microsoft.Azure.SignalR.Protocol.JoinGroupWithAckMessage waiting for ack timed out.

System.TimeoutException: at Microsoft.Azure.SignalR.ServiceConnectionContainerBase+d58.MoveNext (Microsoft.Azure.SignalR.Common, Version=1.6.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.SignalR.MultiEndpointServiceConnectionContainer+<>cDisplayClass24_0+<b0>d.MoveNext (Microsoft.Azure.SignalR.Common, Version=1.6.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.SignalR.MultiEndpointServiceConnectionContainer+<>cDisplayClass29_0+<b2>d.MoveNext (Microsoft.Azure.SignalR.Common, Version=1.6.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.SignalR.MultiEndpointServiceConnectionContainer+d24.MoveNext (Microsoft.Azure.SignalR.Common, Version=1.6.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.SignalR.ServiceLifetimeManagerBase1+<WriteCoreAsync>d__271.MoveNext (Microsoft.Azure.SignalR, Version=1.6.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at SignalRWebApp.SignalRHub.RealtimeHub+d__8.MoveNext (SignalRWebApp, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullSignalRWebApp, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: D:\a\1\s\SignalRWebApp\SignalRHub\RealtimeHub.csSignalRWebApp, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 127)

Further technical details

vicancy commented 3 years ago

Could it be that the Azure SignalR Service is overloaded?

Sounds possible What Unit is your resource? FYI here is a performance case study for tiny group https://docs.microsoft.com/en-us/azure/azure-signalr/signalr-concept-performance#small-group You can also email me your resource ID for me to have a further check.