search

dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.
https://asp.net
MIT License
35.5k stars 10.04k forks source link

Retry ConnectionClosedWhenResponseDoesNotSatisfyMinimumDataRate #58169

Closed sebastienros closed 1 month ago

sebastienros commented 1 month ago

Failing Test(s)

Error Message

2024-09-30T13:07:37.6503447Z The active Test Run was aborted because the host process exited unexpectedly. Please inspect the call stack above, if available, to get more information about where the exception originated from.
2024-09-30T13:07:37.6503680Z The test running when the crash occurred: 
2024-09-30T13:07:37.6503986Z Microsoft.AspNetCore.Server.Kestrel.Sockets.FunctionalTests.ResponseTests.ConnectionClosedWhenResponseDoesNotSatisfyMinimumDataRate
2024-09-30T13:07:37.6504110Z Microsoft.AspNetCore.Server.Kestrel.Sockets.FunctionalTests.MaxRequestBufferSizeTests.LargeUpload

Stacktrace

```text [6.060s] [Microsoft.AspNetCore.Server.Kestrel.BadRequests] [Debug] Connection id "0HN719QBKA5AK" request processing ended abnormally. Microsoft.AspNetCore.Connections.ConnectionAbortedException: The connection was timed out by the server because the response was not read by the client at the specified minimum data rate. at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result) at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application) in /_/src/Servers/Kestrel/Core/src/Internal/Http/HttpProtocol.cs:line 583 ```

Logs

```text [0.001s] [TestLifetime] [Information] Starting test ConnectionClosedWhenResponseDoesNotSatisfyMinimumDataRate-False at 2024-09-30T12:51:46 [0.003s] [Microsoft.Extensions.Hosting.Internal.Host] [Debug] Hosting starting [0.005s] [Microsoft.Hosting.Lifetime] [Information] Now listening on: http://127.0.0.1:63563 [0.006s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Debug] Loaded hosting startup assembly Sockets.FunctionalTests [0.006s] [Microsoft.Hosting.Lifetime] [Information] Application started. Press Ctrl+C to shut down. [0.006s] [Microsoft.Hosting.Lifetime] [Information] Hosting environment: Production [0.006s] [Microsoft.Hosting.Lifetime] [Information] Content root path: C:\h\w\9D7B0886\w\B8AD09B4\e\ [0.006s] [Microsoft.Extensions.Hosting.Internal.Host] [Debug] Hosting started [0.006s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] TestServer is listening on port 63563 [0.006s] [Microsoft.AspNetCore.Server.Kestrel.Sockets.FunctionalTests.ResponseTests.ConnectionClosedWhenResponseDoesNotSatisfyMinimumDataRate] [Information] Sending request [0.007s] [Microsoft.AspNetCore.Server.Kestrel.Sockets.FunctionalTests.ResponseTests.ConnectionClosedWhenResponseDoesNotSatisfyMinimumDataRate] [Information] Sent request [0.007s] [Microsoft.AspNetCore.Server.Kestrel.Sockets.FunctionalTests.ResponseTests.ConnectionClosedWhenResponseDoesNotSatisfyMinimumDataRate] [Information] Waiting for connection to abort. [0.008s] [Microsoft.AspNetCore.Server.Kestrel.Connections] [Debug] Connection id "0HN719QBKA5AK" accepted. [0.008s] [Microsoft.AspNetCore.Server.Kestrel.Connections] [Debug] Connection id "0HN719QBKA5AK" started. [0.008s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request starting HTTP/1.1 GET http:/// - null null [0.008s] [App] [Information] Request received [6.039s] [Microsoft.AspNetCore.Server.Kestrel.BadRequests] [Debug] Connection id "0HN719QBKA5AK", Request id "0HN719QBKA5AK:00000001": the connection was closed because the response was not read by the client at the specified minimum data rate. [6.039s] [Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets] [Debug] Connection id "0HN719QBKA5AK" sending RST because: "The connection was timed out by the server because the response was not read by the client at the specified minimum data rate." [6.039s] [Microsoft.AspNetCore.Server.Kestrel.Connections] [Debug] Connection id "0HN719QBKA5AK" disconnecting. [6.059s] [App] [Information] Wrote 2369536 bytes [6.059s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "0HN719QBKA5AK", Request id "0HN719QBKA5AK:00000001": The request was aborted by the client. [6.060s] [Microsoft.AspNetCore.Hosting.Diagnostics] [Information] Request finished HTTP/1.1 GET http:/// - 200 268435456 null 6051.2758ms [6.060s] [Microsoft.AspNetCore.Server.Kestrel.BadRequests] [Debug] Connection id "0HN719QBKA5AK" request processing ended abnormally. Microsoft.AspNetCore.Connections.ConnectionAbortedException: The connection was timed out by the server because the response was not read by the client at the specified minimum data rate. at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result) at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application) in /_/src/Servers/Kestrel/Core/src/Internal/Http/HttpProtocol.cs:line 583 [6.060s] [Microsoft.AspNetCore.Server.Kestrel.Connections] [Debug] Connection id "0HN719QBKA5AK" stopped. [6.075s] [Microsoft.AspNetCore.Server.Kestrel.Sockets.FunctionalTests.ResponseTests.ConnectionClosedWhenResponseDoesNotSatisfyMinimumDataRate] [Information] Connection was aborted after 6067ms. [6.075s] [Microsoft.Extensions.Hosting.Internal.Host] [Debug] Hosting stopping [6.075s] [Microsoft.Hosting.Lifetime] [Information] Application is shutting down... [6.076s] [Microsoft.Extensions.Hosting.Internal.Host] [Debug] Hosting stopped [6.097s] [TestLifetime] [Information] Finished test ConnectionClosedWhenResponseDoesNotSatisfyMinimumDataRate-False in 6.0957895s ```

Build

https://dev.azure.com/dnceng-public/public/_build/results?buildId=821995 and https://dev.azure.com/dnceng-public/public/_build/results?buildId=821050

sebastienros commented 1 month ago

The builds mention two tests were running while they failed (MaxRequestBufferSizeTests.LargeUpload), the other test is already retried. If this continues to fail I think we should quarantine MaxRequestBufferSizeTests.LargeUpload to see if this is the actual reason.