A lot of time spent in Result execution with eventual exception in kestrel

[senioroman4uk]

Hi, I was trying to diagnose an issue that we observe for a small population of slow failing requests in production (less then 1%). Our service is hosted on Kubernetes Linux (Azure Linux V2). From global traffic routing we use Azure Traffic Manager, so clients connect to the service directly.

We use mariner base images mcr.microsoft.com/dotnet/aspnet:8.0-cbl-mariner2.0

dotnet --info

Host:
  Version:      8.0.8
  Architecture: x64
  Commit:       08338fcaa5
  RID:          linux-x64

.NET SDKs installed:
  No SDKs were found.

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.8 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.8 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

We log duration of various parts of MVC pipeline via filters (action, result and resource filter) and also measure an entire request duration via middleware. For these slow requests we observe a lot of time spent in result execution (for example 969520 ms) with an eventual exception from kestrel

 at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.IO.Pipelines.StreamPipeWriter.FlushAsyncInternal(Boolean writeToStream, ReadOnlyMemory`1 data, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.IO.Pipelines.StreamPipeWriter.CompleteAsync(Exception exception)
   at System.IO.Pipelines.StreamPipeWriter.CompleteAsync(Exception exception)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.DuplexPipeStreamAdapter`1.DisposeAsync()
   at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware.OnConnectionAsync(ConnectionContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection`1.ExecuteAsync()
InnerException type:System.Net.Sockets.SocketException message:No route to host stack:   at System.IO.Pipelines.Pipe.WriteAsync(ReadOnlyMemory`1 source, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.DuplexPipeStream.WriteAsync(ReadOnlyMemory`1 source, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)

I would like to get some help identifying the root cause of this issue

[captainsafia]

@senioroman4uk Can you share more details about whether or not this bottleneck occurs for a specific result type? I know that getting minimal repros for these kinds of issues can be hard but additional info about when it happens would be appreciated.

[dotnet-policy-service[bot]]

Hi @senioroman4uk. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[dotnet-policy-service[bot]]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.

See our Issue Management Policies for more information.