search

dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.
https://asp.net
MIT License
35.43k stars 10.02k forks source link

ASP.NET Core 3.0 Blazor Server fails to load when deploying multiple servers #16645

Closed pheuter closed 5 years ago

pheuter commented 5 years ago

Versions

Startup Configuration

services.AddSignalR().AddAzureSignalR();
services.AddRazorPages();
services.AddServerSideBlazor();

Error

In the browser console:

Error: The list of component records is not valid.
Uncaught (in promise) Error: Invocation canceled due to the underlying connection being closed.

Additional context

This error only occurs when I deploy the ASP.NET Core app to multiple servers behind a load balancer. The docs seem to suggest that when using Azure SignalR Service, sticky sessions are handled automatically for you, but this error appears to suggest otherwise?

danroth27 commented 5 years ago

@bradygaster @anurse

danroth27 commented 5 years ago

@pheuter Can you tell us more about your environment? Which load balancer are you using? Are the servers and load balancer on Azure or in a local environment?

pheuter commented 5 years ago

Sure thing, @danroth27

3 servers is misleading. The actual environment is a Kubernetes deployment of replica size 3 on an AKS cluster, so it's really 3 pods potentially running across 3 VMs in a VNet managed by the AKS cluster. The load balancer is provisioned automatically on Azure using a Kubernetes Nginx ingress controller.

analogrelay commented 5 years ago

Could you enable tracing on the client side? We have docs on how to do this in SignalR here: https://docs.microsoft.com/en-us/aspnet/core/signalr/diagnostics?view=aspnetcore-3.0 . I believe there's a way to access the necessary config options in Blazor but I don't know it off-hand (@pranavkm ?).

Uncaught (in promise) Error: Invocation canceled due to the underlying connection being closed.

This indicates that the connection is being closed abruptly, possible a proxy.

The load balancer is provisioned automatically on Azure using a Kubernetes Nginx ingress controller.

Is Nginx configured to properly proxy WebSockets? When I see an abrupt closure like this, the first thing I think is proxies :)

pheuter commented 5 years ago

@anurse I will setup the tracing shortly, just want to further clarify that everything works as expected when I simply change the Kubernetes deployment replica size to 1. This means that requests to that single pod are still routed through the same load balancer and nginx ingress controller, and the wss:// connection appears to work just fine.

analogrelay commented 5 years ago

everything works as expected when I simply change the Kubernetes deployment replica size to 1. This means that requests to that single pod are still routed through the same load balancer and nginx ingress controller, and the wss:// connection appears to work just fine.

That does generally indicate the proxy is working fine. Thanks for that reminder of the context you provided above :). Let's get some tracing and see what that yields.

pranavkm commented 5 years ago

Here's how you configure the SignalR client that Blazor uses: https://docs.microsoft.com/en-us/aspnet/core/blazor/hosting-models?view=aspnetcore-3.0#configure-the-signalr-client-for-blazor-server-apps

pheuter commented 5 years ago

Server-side logs:

dbug: Microsoft.AspNetCore.SignalR.Internal.DefaultHubDispatcher[1]
      Received hub invocation: InvocationMessage { InvocationId: "0", Target: "StartCircuit", Arguments: [ https://web.careswitch.com/, https://web.careswitch.com/login, [{"type":"server","sequence":0,"descriptor":"CfDJ8HBfOAK5iB9BiwSknWLLLPOeUDbgczl8pIbnhvcxj7QK-tKBIX3UjpzZAlAQd2-ptoPQswOOQHwosSy91nyi7NID_iAnuqy2FhG52glnEm-GyOj__AYIPz19MU188jpmofmGOgea2_uX8tGtzfVOUE0pNXvMpGfH6GY1Y312GW6Nl5px3MyFV7NACC6RxfNwmIB83og2_X-NSZLyUz7YGryKBcvHAId1mP2Ffg796OJ-6Hd7eLY7gPJsqpzT5oL8KxqJ5UN6LiH2zVBhKSUr88AKdBZzJta0VgSEDXEpENMe"}] ], StreamIds: [  ] }.

Client-side logs

[Error] Failed to load resource: the server responded with a status of 400 () (disconnect, line 0)
[Info] [2019-10-30T00:48:41.261Z] Information: Normalizing '_blazor' to 'https://web.careswitch.com/_blazor'. (blazor.server.js, line 1)
[Log] [2019-10-30T00:48:41.261Z] Debug: Starting HubConnection. (blazor.server.js, line 1)
[Log] [2019-10-30T00:48:41.262Z] Debug: Starting connection with transfer format 'Binary'. (blazor.server.js, line 1)
[Log] [2019-10-30T00:48:41.262Z] Debug: Sending negotiation request: https://web.careswitch.com/_blazor/negotiate. (blazor.server.js, line 1)
[Log] [2019-10-30T00:48:41.300Z] Debug: Sending negotiation request: https://careswitch-web.service.signalr.net/client/negotiate?hub=componenthub&asrs.op=%2F_blazor&asrs_request_id=O%2BVldtCqFAA%3D. (blazor.server.js, line 1)
[Log] [2019-10-30T00:48:41.341Z] Debug: Selecting transport 'WebSockets'. (blazor.server.js, line 1)
[Info] [2019-10-30T00:48:41.447Z] Information: WebSocket connected to wss://careswitch-web.service.signalr.net/client/?hub=componenthub&asrs.op=%2F_blazor&asrs_request_id=O%2BVldtCqFAA%3D&id=Kt0js0GFTRqcQVwzZDzoUgd58b758b1&access_token=eyJhbGciOiJIUzI1NiIsImtpZCI6IjA2a2tHUVkwVHg1K3hvZHR3TUd2T0F4MFpqZWJWMVBpdGNiTmdkL1ZXL1E9IiwidHlwIjoiSldUIn0.eyJhc3JzLnMuaWQiOiJvN1pqZHRDcUZBQT0iLCJuYmYiOjE1NzIzOTY1MjEsImV4cCI6MTU3MjQwMDEyMSwiaWF0IjoxNTcyMzk2NTIxLCJhdWQiOiJodHRwczovL2NhcmVzd2l0Y2gtd2ViLnNlcnZpY2Uuc2lnbmFsci5uZXQvY2xpZW50Lz9odWI9Y29tcG9uZW50aHViIn0.Q37VDB6ZclJVvrKD26l9NhebAdOBTNmOz-ipD0hhv3I. (blazor.server.js, line 1)
[Log] [2019-10-30T00:48:41.447Z] Debug: The HttpConnection connected successfully. (blazor.server.js, line 1)
[Log] [2019-10-30T00:48:41.448Z] Debug: Sending handshake request. (blazor.server.js, line 1)
[Info] [2019-10-30T00:48:41.448Z] Information: Using HubProtocol 'blazorpack'. (blazor.server.js, line 1)
[Log] [2019-10-30T00:48:41.529Z] Debug: Server handshake complete. (blazor.server.js, line 1)
[Log] [2019-10-30T00:48:41.529Z] Debug: HubConnection connected successfully. (blazor.server.js, line 1)
[Error] [2019-10-30T00:48:41.589Z] Error: The list of component records is not valid.
    (anonymous function) (blazor.server.js:15:27319)
    S (blazor.server.js:8:104321)
    (anonymous function) (blazor.server.js:8:103847)
    forEach
    (anonymous function) (blazor.server.js:1:19127)
    (anonymous function) (blazor.server.js:1:17184)
    (anonymous function) (blazor.server.js:1:38037)
[Log] [2019-10-30T00:48:41.589Z] Debug: Stopping HubConnection. (blazor.server.js, line 1)
[Log] [2019-10-30T00:48:41.590Z] Debug: HttpConnection.stopConnection(undefined) called while in state Disconnecting. (blazor.server.js, line 1)
[Info] [2019-10-30T00:48:41.590Z] Information: Connection disconnected. (blazor.server.js, line 1)
[Log] [2019-10-30T00:48:41.590Z] Debug: HubConnection.connectionClosed(undefined) called while in state Disconnecting. (blazor.server.js, line 1)
[Error] Unhandled Promise Rejection: Error: Invocation canceled due to the underlying connection being closed.
    (anonymous function) (blazor.server.js:8:100661)
    a (blazor.server.js:8:99498)
    promiseReactionJob

Possibly also worth noting that it works roughly a third of the time, which suggests that it doesn't like it when the WebSocket connection goes to a server other than the one that prerendered the initial view.

pheuter commented 5 years ago

Not sure if it's related but I'm also loading Blazor.Polyfill before blazor.server.js for compatibility with IE11.

javiercn commented 5 years ago

@pheuter Thanks for contacting us.

From the error message I believe you don't have stickiness configured both on the Kubernetes Ingress

javiercn commented 5 years ago

See https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/ for how to enable affinity in the ingress controller and https://github.com/Azure/azure-signalr/blob/dev/docs/use-signalr-service.md#serverstickymode for details.

pheuter commented 5 years ago

Thanks for the suggestion @javiercn, looks like you might be right. I'm going to give it a shot and report back,

Maybe this is just me not paying attention but the ASP.NET Core Blazor docs could probably make it clearer that there is additional setup involved to get sticky sessions working. The only documentation I found was this section that just talks about Azure App Service. And then this section talks about how when using Azure SignalR Service:

sticky sessions, also known as client affinity, is not required, because clients are immediately redirected to the Azure SignalR Service when they connect.

...which seems misleading.

pheuter commented 5 years ago

I can confirm it works now after I made the following changes:

Startup.cs

services.AddSignalR().AddAzureSignalR(options =>
{
    options.ServerStickyMode = Microsoft.Azure.SignalR.ServerStickyMode.Required;
});

Kubernetes Ingress definition

nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "affinity"
nginx.ingress.kubernetes.io/session-cookie-expires: "14400"
nginx.ingress.kubernetes.io/session-cookie-max-age: "14400"
javiercn commented 5 years ago

@pheuter I'm glad that it worked out.

I'm closing this issue as the problem has been solved. I've filed an doc issue here https://github.com/aspnet/AspNetCore.Docs/issues/15392 to improve the documentation