Getting random WebSocket failed connection of Blazor Server Application hosted on an Azure App Service

fontainerc commented 5 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the bug

I have a Blazor Server Application which is hosted on an Azure App Service that has a random connection error. When it occurs, the following errors are in the browser console window:

WebSocket connection to 'website/_blazor?id=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' failed:
Information: (WebSockets transport) There was an error with the transport
Error: Failed to start the transport 'WebSockets': Error: WebSocket failed to connect. The connection could not be found on the server, either the endpoint may not be a SignalR endpoint, the connection ID is not preseent on the server, or there is a proxy blocking WebSockets. If you have multiple servers check that sticky sessions are enabled.
Failed to load resource: the server responded with a status of 404 (Nout Found)
Error: Connection disconnected with error 'Error: No Connection with that ID: Status code '404'
Warning: Failed to connect via WebSockets, using the Long Polling fallback transport. This may be due to a VPN or proxy blocking the connection
Error: Failed to start the circuit

Expected Behavior

Blazor Server Application running on an Azure App Service with SignalR should run without random loading errors.

Steps To Reproduce

Here is the code from Program.cs file, if it helps:

using BlazorServerApp.Data; using Microsoft.AspNetCore.Components; using Microsoft.AspNetCore.Components.Web; using Microsoft.AspNetCore.Connections; using Microsoft.EntityFrameworkCore; using System;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddSignalR();

builder.Services.AddRazorPages(); builder.Services.AddServerSideBlazor();

var app = builder.Build();

// Configure the HTTP request pipeline. if (!app.Environment.IsDevelopment()) { app.UseExceptionHandler("/Error"); // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts. app.UseHsts(); }

app.UseHttpsRedirection();

app.UseStaticFiles();

app.UseRouting();

app.MapBlazorHub(); app.MapFallbackToPage("/_Host");

app.Run();

Exceptions (if any)

No response

.NET Version

.NET 6.0

Anything else?

No response

dmm-l-mediehus commented 5 months ago

Did you try enabling WebSockets in the Azure options? Web App -> Settings -> Configuration

fontainerc commented 5 months ago

Yes, it is enabled and always on.

This is an erratic issue, it works 90% of the time and 10% of the time this error occurs.

I am hoping that there is a way to intensify in code when this error occurs and then re-establish the SignalR connection with the correct ID without sending a message to through the browser.

cc: @guardrex https://github.com/dotnet/AspNetCore.Docs/issues/32470

mkArtakMSFT commented 5 months ago

Thanks for contacting us. 10% is quite large and concerning number. And this doesn't sound related to the Blazor itself. So this seems to be more of an issue with the environment you're running in. To help figuring this out we just have a set of questions at this point. Things like firewalls, etc.?

What if this is really a server issue and for whatever reason it's being recycled that often? Did you try to engage with AppService if that's the case?
Have you tried to upgrade to .NET 8 (just for testing) and see if you observe the same behavior or not?
As a side note, it seems that you're observing these issues yourself. Is this something other customers also impacted with?

Bottom line is that these types of issues can happen for variety of reasons and there is no simple answer to tell what's causing these. So you'll have to work through these questions to understand what is the real cause and fix that.

fontainerc commented 5 months ago

Thanks for your feedback. I will answer your questions as best as I can. Note that my 10% number is a qualitative observation and is not based on actual data. My business and website is rather new, so I haven't gotten much client feedback yet.

I don't have a firewall configured with my Azure App Service.
I checked the Activity log and the server does not appear to be recycling when the issue occurs.
I have not tried upgrading to .NET 8 yet, but that is a good suggestion. I will try to do that in test mode to see if that helps. I will post the results here.
You are correct that I have been observing these issues mostly myself. On my website launch day last Wednesday (4/24/24), I had 14 users at one point and it did appear there were some failures associated with those users during that time so it is a concern of mine.

Is there a way to refresh the SignalR ID within the Blazor app when this occurs? I think it is related to trying to reconnect within the same browser instance when a period of time passes or when I connect to it on my phone.

I appreciate any help you can provide! Thanks!

mkArtakMSFT commented 5 months ago

Thanks for additional details. Based on your answers 0 & 1 it seems what you're experiencing is client connectivity issues, which can happen because of multiple reasons. Phones, for example, have custom logic to drop connections which are not actively used to save on resources. That can be one of the reasons.

As for a better control over the reconnection, we do plan to improve the current reconnection experience as part of the https://github.com/dotnet/aspnetcore/issues/32113

That said, maybe you still will get a better experience with .NET 8, as I don't even remember if we've made any changes since 6 in this area.

boreys commented 2 months ago

I'm getting this kind of error with dotnet 8 when running Blazor Web App behind load balancer with two or more app instance. It's not consistent error. When it failed with 404 error, I had to refresh the browser for it to work again. Session affinity is enable and everything is working, it's just that sometime it failed. I cannot find documentation about the Blazor.start javascript client side so that I can add code to auto refresh when it fail.

CodeFontana commented 2 months ago

I'm also experiencing this issue on .NET 8, Blazor Web App. My landing page is a static server-side page, however, most of my features use an interactive server-side page.

When navigating to any of the interactive pages, not necessarily starting from the static page, could be interactive to interactive, intermittently, I receive the errors/output described on this issue.

Interestingly, I have 3x environments. I can easily reproduce in two of them, but one is working solid. I've compared the Configurations and App Service Plans, between working and non-working, to no avail.

Here's a video, reproducing it. It happened to occur on the first attempt, and I navigated back to the static home page, and back to the interactive page, and everything was fine: https://github.com/user-attachments/assets/eb542cba-d6b9-4deb-9d9f-6a44b0229fc7

And for reference, here are my Configurations for this App Service. It is a P1v2...

Any suggestions would be very much appreciated. In the future, I'm likely to choose the WebAssembly + API approach. I've always had issues with Blazor Server, whether hosted on-prem or in Azure. It's very disheartening for something that is so close to being amazing.

demayer commented 1 month ago

Same issue here. We deployed the App on a kubernetes cluster with 3 pods hosting the app behind an nginx reverse proxy and load balancer. If 2 pods are shut down, everything works as expected.

I guess configuring nginx to handle sticky sessions for the websocket connections would solve the problem. (not possible in our case due to constraints related to the hosting environment)
The other solution of having a backplane like Redis to handle sessions sounds overly complicated with poor guidance in the docs.

Any help would be appreciated, as the backplane path looks like the only option with little to no documentation for a Blazor Web App.

clrockwell commented 1 month ago

I'm seeing the same thing. I created a SO post for it: https://stackoverflow.com/questions/78915746/deploying-blazor-app-to-azure-linux-app-service-seemingly-random-web-socket-fa

It's a pretty simple app, hosted on an Azure App Service, Linux, B1 plan.

<PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <UserSecretsId>foo</UserSecretsId>
  </PropertyGroup>

  <PropertyGroup Condition=" '$(RunConfiguration)' == 'https' " />
  <PropertyGroup Condition=" '$(RunConfiguration)' == 'http' " />
  <ItemGroup>
    <PackageReference Include="Azure.Messaging.ServiceBus" Version="7.18.1" />
    <PackageReference Include="Microsoft.EntityFrameworkCore" Version="8.0.8" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="8.0.8" />
    <PackageReference Include="Microsoft.Extensions.Azure" Version="1.7.5" />
    <PackageReference Include="MudBlazor" Version="7.6.0" />
  </ItemGroup>

</Project>

CodeFontana commented 1 month ago

This issue was resolved via following the guidance here: https://learn.microsoft.com/en-us/aspnet/core/blazor/fundamentals/signalr?view=aspnetcore-8.0

Specifically, we needed to turn on Session Affinity in the App Service:

As it turns out, we have two lower environments. One did not require session affinity, but the other one absolutely did. I believe one of the environments uses a load balancer, and the other did not... which created the confusion.

clrockwell commented 1 month ago

This issue was resolved via following the guidance here: https://learn.microsoft.com/en-us/aspnet/core/blazor/fundamentals/signalr?view=aspnetcore-8.0

Thanks; it appears to have solved my issue as well. I must've glossed over it before as I checked my app service for this setting and didn't see it.

jacovisage commented 1 month ago

Switching the Session affinity to On resolved my issue aswell.

guardrex commented 1 month ago

UPDATED: I've researched this subject, and I'll be loading up a docs PR to clarify the state of affairs.

Please post further discussion comments on the docs issue ...

https://github.com/dotnet/AspNetCore.Docs/issues/33591

zvasilius commented 1 month ago

Have the same issue with .NET 8 blazor server (hosted on IIS). Any ideas how to solve it for IIS case?

cpajchambers commented 1 month ago

I believe I have the same problem with an app service running a SSR Blazor app. The web socket connection sometimes fails and the app falls back to long polling. My app is hosted on Linux and on just one instance. I have the same app running in two identical app services - one of them (my test environment) is absolutely fine and never has this problem, but the other (production :S) does.

I tried enabling session affinity but it doesn't help. I presume this is because I only have one instance. I have a meeting scheduled with Microsoft support this week so, if I get to the bottom of the problem, I'll report back.

CodeFontana commented 1 month ago

Have the same issue with .NET 8 blazor server (hosted on IIS). Any ideas how to solve it for IIS case?

Do you have this role installed on your IIS server? Web Server (IIS) --> Web Server --> Application Development --> Web Socket Protocol

zvasilius commented 1 month ago

Have the same issue with .NET 8 blazor server (hosted on IIS). Any ideas how to solve it for IIS case?

Do you have this role installed on your IIS server? Web Server (IIS) --> Web Server --> Application Development --> Web Socket Protocol

Yes, I have Web Socket Protocol installed. The interesting part here is that works ok for some clients, but many have exactly the same error in Browser console. I was assuming it might be due to some vpn/proxy between, but users claimed there is no such.

dotnet / aspnetcore