Sending SignalR messages through the ServiceHubContext sometimes produces 403

[ryanheath]

Describe the bug

Sending messages through the ServiceHubContext sometimes produces 403. We are using Azure managed identity for the Azure function that is posting the messages to SignalR hub. We see periodic failures in between half hours. But mostly the exact same calls are working perfectly fine. What could we look at to fix this problem?

To Reproduce

Not really a way to reproduce it.

await _signalRHubContextProvider.xxx.Clients.Group(sendMessageRequest.GroupName)
    .SendCoreAsync(sendMessageRequest.Target,
        new object?[]
        {
            payloadObj
        },
        cancellationToken);

Exceptions

Exception while executing function: Functions.SendMessage Result: Failure
Exception: System.AggregateException: One or more errors occurred. (Azure SignalR service runtime error. Request Uri: https://xxx.service.signalr.net/api/hubs/xxx/groups/xxx/:send?api-version=2022-06-01)
 ---> Microsoft.Azure.SignalR.Common.AzureSignalRRuntimeException: Azure SignalR service runtime error. Request Uri: https://xxx.service.signalr.net/api/hubs/xxx/groups/xxx/:send?api-version=2022-06-01
 ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 403 (Forbidden)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.SignalR.RestClient.ThrowExceptionOnResponseFailureAsync(HttpResponseMessage response)
   at Microsoft.Azure.SignalR.RestClient.SendAsync(RestApiEndpoint api, HttpMethod httpMethod, String productInfo, String methodName, Object[] args, Func`2 handleExpectedResponseAsync, CancellationToken cancellationToken)
   at Microsoft.Azure.SignalR.Management.RestHubLifetimeManager`1.SendGroupExceptAsync(String groupName, String methodName, Object[] args, IReadOnlyList`1 excludedConnectionIds, CancellationToken cancellationToken)

Further technical details

<PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.18.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.0.13" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.SignalRService" Version="1.10.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.13.0" />

[terencefan]

Doesn't look like a SDK side bug. Please send email to vscsignalr@microsoft.com along with your resourceId and issue description for help.

[terencefan]

A possible reason is the internal request took longer than expected, which results in 403 error. Do you know how long those failed requests took and the precise timestamp? It may help us locate the problem.

[ryanheath]

The failing requests are failing fast too, usually 2 digit ms.

[ryanheath]

Doesn't look like a SDK side bug. Please send email to vscsignalr@microsoft.com along with your resourceId and issue description for help.

I have sent an email with relevant information.

Thanks! // Ryan

[terencefan]

The incident should have been mitigated on 5/28, ~5AM

[ryanheath]

We see indeed zero errors now!

Thanks // Ryan