"Unexpected end of request content" preventing terminating chunk from being sent

[stephentoub]

More details here: https://github.com/dotnet/corefx/issues/17330#issuecomment-288078562

In summary, some clients (e.g. libcurl) stop sending request content if they concurrently receive an error status code in response from the server. If an app on kestrel ignores the request data and immediately sends a response with an error status, the client may as such stop sending the request data, which in turn causes kestrel to fail with a BadHttpRequestException, but it doesn't complete the chunked encoding response by sending a terminating 0-length chunk, nor does it close the connection. As a result, the client is left waiting indefinitely for the remainder of the response that never comes, and effectively deadlocks until timeout.

[davidfowl]

Were you using 1.1?

[stephentoub]

Yes:

  <PropertyGroup>
    <TargetFramework>netcoreapp1.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Folder Include="wwwroot\" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore" Version="1.1.1" />
  </ItemGroup>

[cesarblum]

Reading through the other issue, will have an answer soon.

[cesarblum]

Self-assigning for investigation. @stephentoub see https://github.com/dotnet/corefx/issues/17330#issuecomment-288168960

[cesarblum]

I thought we'd shipped "FIN as connection reset" in Kestrel 1.1 but we didn't. This was the change: https://github.com/aspnet/KestrelHttpServer/pull/1218

The current behavior is bad though, in that the server at least do something. Given 1.1 doesn't close the connection after the client FIN, it should attempt to complete the response. This may be a candidate for a patch, but I need to assess how large the change would be. cc @muratg @Eilon

[muratg]

Filed https://github.com/aspnet/KestrelHttpServer/issues/1535 to track this for 1.1.2 patch release.

[cesarblum]

Diagnosis here: https://github.com/dotnet/corefx/issues/17330#issuecomment-288248663

[lewislabs]

We're still seeing this error when using Kestrel 1.1.2. Getting a BadHttpRequestException with the message Unexpected end of request content being thrown with this StackTrace.

at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.Frame.RejectRequest(BadHttpRequestException ex)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.MessageBody.ForContentLength.PeekAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.MessageBody.ReadAsync(ArraySegment`1 buffer, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.FrameRequestStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.FrameRequestStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.StreamReader.ReadBuffer()
   at System.IO.StreamReader.ReadToEnd()

It;s not something that I've been able to easily reproduce but I'm looking for any advice that anyone has on it.

[cesarblum]

@lewislabs The original issue only happened on requests that used the chunked transfer encoding. From your stack trace, I can see that the request includes a Content-Length header (otherwise Kestrel wouldn't be using the ForContentLength class to read the request body.

Could the client have disconnected mid-request?

[lewislabs]

That's certainly possible but it would be unintentional - we're in control of both the client side and server side code. We're seeing this error fairly frequently but it's only for a small fraction of requests. Would a client disconnection be the most likely cause of that stack trace?

[cesarblum]

It would definitely show up if the client disconnected before sending the entire request body. You can look for client disconnects in the logs. Looks for messages about ECONNRESET and FIN on the connections where you're seeing that stack trace.

Another way to look into what's happening would be to use a tool like Wireshark to see what's getting through the wire.

[maroallegro]

@lewislabs @cesarbs Did you solve the issue? I am also having this problem at the moment. I am trying to send large file (20GB) from WebAPI1 to WebAPI2 (hosted locally). When I am sending this file via postman to WebAPI2 it sends whole file. But when I am trying to send the file from WebAPI1 to WebAPI2 it fails (I am able to send files like 7GB though).

During sending 20GB file I am receiving error:

Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected
 end of request content.
   at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.PipeCompl
etion.ThrowFailed()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.Pipe.GetR
esult(ReadResult& result)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.Pipe.Micr
osoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.IReadableBufferAwai
ter.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.System.IO.Pipelines.ReadableB
ufferAwaitable.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody.<ReadAs
ync>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNot
ification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.FrameRequestStream.
<ReadAsyncInternal>d__21.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNot
ification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.AspNetCore.WebUtilities.BufferedReadStream.<EnsureBufferedAsync>
d__37.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNot
ification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.<ReadAsync>d__36.M
oveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNot
ification(Task task)
   at System.IO.Stream.<CopyToAsyncInternal>d__27.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNot
ification(Task task)

[cesarblum]

@maroallegro What do you see on the WebAPI1 side?

[maroallegro]

@cesarbs @lewislabs When I run this request as part of WCF contract in WebApi1 i got:

   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()

Operation failed always around 2 minutes...

Than I tried to send that 20GB file as part of controller method (from WebAPI1) and catched there TaskCanceledException. When this exception occurs, we can tell was that a timeout issue by checking the flag IsCancellationRequested if it false it was timeout issue. It was timeout in my case, issue was in Timeout of HttpClient: new HttpClient { Timeout = TimeSpan.FromMinutes(2) }; increasing the limit resolved the issue.