System.Net.Sockets.SocketException (125): Operation canceled

j4587698 commented 4 years ago

Describe the bug

call Microsoft Account external login setup with nginx proxy fails(502 Bad Gateway). Local debugging and not using nginx proxy are success.

logs:

warn: Microsoft.AspNetCore.Server.Kestrel[0]
      Connection processing ended abnormally.
System.Net.Sockets.SocketException (125): Operation canceled
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.<GetResult>g__ThrowSocketException|7_0(SocketError e)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.ProcessReceives()
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.DoReceive()
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token)
   at System.IO.Pipelines.Pipe.DefaultPipeReader.ReadAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.BeginRead(ValueTask`1& awaitable)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)

nginx configuration

location / {
       proxy_pass http://localhost:5000;
       proxy_http_version 1.1;
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection keep-alive;
       proxy_set_header Host $host;
       proxy_cache_bypass $http_upgrade;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header X-Forwarded-Proto $scheme;
     }

Further technical details

ASP.NET Core version 3.1.3 and 5.0

Include the output of dotnet --info


..NET Core SDK（反映任何 global.json）:
Version:   5.0.100-preview.4.20258.7
Commit:    65f0fc2cad

运行时环境: OS Name: Windows OS Version: 6.1.7601 OS Platform: Windows RID: win7-x64 Base Path: C:\Program Files\dotnet\sdk\5.0.100-preview.4.20258.7\

Host (useful for support): Version: 5.0.0-preview.4.20251.6 Commit: 47ec733ba7

.NET SDKs installed: 3.1.300-preview-015135 [C:\Program Files\dotnet\sdk] 3.1.300 [C:\Program Files\dotnet\sdk] 5.0.100-preview.2.20176.6 [C:\Program Files\dotnet\sdk] 5.0.100-preview.4.20258.7 [C:\Program Files\dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.All 2.1.16 [C:\Program Files\dotnet\shared\Microsoft.AspN etCore.All] Microsoft.AspNetCore.App 2.1.16 [C:\Program Files\dotnet\shared\Microsoft.AspN etCore.App] Microsoft.AspNetCore.App 3.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNe tCore.App] Microsoft.AspNetCore.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNe tCore.App] Microsoft.AspNetCore.App 5.0.0-preview.4.20257.10 [C:\Program Files\dotnet\sha red\Microsoft.AspNetCore.App] Microsoft.NETCore.App 2.1.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore .App] Microsoft.NETCore.App 3.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore. App] Microsoft.NETCore.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore. App] Microsoft.NETCore.App 5.0.0-preview.4.20251.6 [C:\Program Files\dotnet\shared\ Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.2 [C:\Program Files\dotnet\shared\Microsoft.W indowsDesktop.App] Microsoft.WindowsDesktop.App 3.1.4 [C:\Program Files\dotnet\shared\Microsoft.W indowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.0-preview.4.20251.1 [C:\Program Files\dotnet\ shared\Microsoft.WindowsDesktop.App]

To install additional .NET runtimes or SDKs: https://aka.ms/dotnet-download


- The IDE (VS / VS Code/ VS4Mac) you're running on, and it's version
VS preview 16.6

ayende commented 3 years ago

Can you explain what the issue was?

dwigand commented 3 years ago

We experienced that after 3 requests Kestrel stopped processing the third request:

warn: Microsoft.AspNetCore.Server.Kestrel[0]
      Connection processing ended abnormally.
System.Net.Sockets.SocketException (125): Operation canceled
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.<GetResult>g__ThrowSocketException|7_0(SocketError e)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.ProcessSends()
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.DoSend()
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult()
   at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)

This exception occured

during the return of the result (the execution of the application logic was already completed)
only with linux containers, but not with linux containers hosted in azure

GiriB commented 3 years ago

Finally we found the issue in our code base: There were some usages of Microsoft.Win32.SafeHandles. We removed them and it works well now.

This was the case of us as well. Removing the reference to SafeHandles stopped the exceptions.

Tratcher commented 3 years ago

Thread summary:

Lots of different people hitting System.Net.Sockets.SocketException (125): Operation canceled in different scenarios
- Kestrel: Causes 502s from NGinx, but also repros without NGinx.
- HttpClient: Not enough info to investigate this variant.
- SqlClient: There were some usages of Microsoft.Win32.SafeHandles. We removed them and it works well now.
Docker or K8 are involved
Windows or Linux
Most reports are from 3.1, before 5.0 was released.
Related to AspNetCore.Identity.LiteDB for some users
No reliable repro
A network capture would be helpful to correlate with timestamped logs.

Tratcher commented 3 years ago

Closing this as non-actionable. There's a wide range of discussion here on issues that are not clearly related. Many of the reports are outdated or from much older versions, and no repro is available.

If anyone is able to reproduce this reliably then please open a new issue with the details.

dotnet / aspnetcore