Azure / azure-signalr

Azure SignalR Service SDK for .NET
https://aka.ms/signalr-service
MIT License
418 stars 100 forks source link

Unable to differentiate 429 response coming from Hub service or ASRS #1102

Open junedans opened 3 years ago

junedans commented 3 years ago

We had an issue where client started to receive 429 response back on trying to connect. Deeper investigation helped us figure that ASRS had reached its peak of concurrent connections and started disconnecting new requests. Raising this issue for ASRS team to make it easy to understand if this response is coming from Hub vs ASRS. This bug investigation required us to dig into ASRS logs to understand what is happening but I think it should be more meaningful if at client SDK level we can figure the same information. Stack trace below for reference: System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 (Too Many Requests). at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode() at Microsoft.AspNetCore.Http.Connections.Client.HttpConnection.<NegotiateAsync>d__44.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Http.Connections.Client.HttpConnection.<GetNegotiationResponseAsync>d__51.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.AspNetCore.Http.Connections.Client.HttpConnection.<SelectAndStartTransport>d__43.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Http.Connections.Client.HttpConnection.<StartAsyncCore>d__40.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Http.Connections.Client.HttpConnection.<StartAsync>d__39.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Http.Connections.Client.HttpConnectionFactory.<ConnectAsync>d__3.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.AspNetCore.Http.Connections.Client.HttpConnectionFactory.<ConnectAsync>d__3.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.SignalR.Client.HubConnection.<StartAsyncCore>d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.SignalR.Client.HubConnection.<StartAsyncInner>d__50.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.SignalR.Client.HubConnection.<StartAsync>d__49.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult()

vwxyzh commented 3 years ago

Hub service will not return 429 normally, unless your app add a middleware to return it. 429 should return from service side, like connections exceed limit, or too many request in a short time.

The problem is we do not give any information about unit size from service runtime. Currently, only service resource provider can return it, it should be an issue.