search

dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.
https://asp.net
MIT License
35.35k stars 9.99k forks source link

BadHttpRequestException: Reading the request body timed out due to data arriving too slowly #34516

Closed jrichardsz closed 3 years ago

jrichardsz commented 3 years ago

Describe the bug

Netcore rest api shows this error when the service is stress tested: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate

To Reproduce

Exceptions (if any)

Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.
   at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken)
   at System.Text.Json.JsonSerializer.ReadAsync[TValue](Stream utf8Json, Type returnType, JsonSerializerOptions options, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
   at Microsoft.AspNetCore.Mvc.Formatters.SystemTextJsonInputFormatter.ReadRequestBodyAsync(InputFormatterContext context, Encoding encoding)
   at Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext bindingContext)
   at Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext actionContext, IModelBinder modelBinder, IValueProvider valueProvider, ParameterDescriptor parameter, ModelMetadata metadata, Object value)
   at Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Further technical details

Precedents

There are two equal issues which were closed :(

BrennanConroy commented 3 years ago

This is an expected error if you're doing stress testing that sends data to the server very slowly. Or if you open too many new connections and overload the server so it can't process them all in time.

This error can be a symptom of thread pool starvation.

shirhatti commented 3 years ago

Can you try running your app with dotnet counters and collecting event counters? That'll help validate our hypotheses on starvation.

The simplest thing to try is: dotnet-counters collect --format json --counters System.Runtime,Microsoft.AspNetCore.Hosting -- dotnet myapp.dll

jrichardsz commented 3 years ago

This is an expected error if you're doing stress testing that sends data to the server very slowly. Or if you open too many new connections and overload the server so it can't process them all in time.

This error can be a symptom of thread pool starvation.

"stress testing that sends data to the server very slowly"... that is not my case. My rest endpoint supports 500 invocations in a sequence mode with an average response time of 1500 milliseconds. But if I try the same endpoint but in parallel mode, does not support even 20 users. More than 20 users, the indicated error begins to appear

If my goal is Set up my c# rest api with optimal parameters to support high user concurrency how Can I increase the thread pool to prevent starvation?

Thanks for your time

jrichardsz commented 3 years ago

Can you try running your app with dotnet counters and collecting event counters? That'll help validate our hypotheses on starvation.

The simplest thing to try is: dotnet-counters collect --format json --counters System.Runtime,Microsoft.AspNetCore.Hosting -- dotnet myapp.dll

I will configure this on my dockerfile. Does this create a file called counter.json?

Thanks for your time

BrennanConroy commented 3 years ago

Does this create a file called counter.json?

Yep, looking forward to the results!

ghost commented 3 years ago

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment. If it is closed, feel free to comment when you are able to provide the additional information and we will re-investigate.

See our Issue Management Policies for more information.