Parallel requests cause responses to get mixed

[portno]

When 2 requests are performed at the same time, there is a possibility that one's response will contain 2 concatenated JSON strings

{"@odata.context":"http://localhost:64149/odata/$metadata#Books(Author())","value":[]}{"@odata.context":"http://localhost:64149/odata/$metadata#Books(Author())","value":[]}

and the other will crash with System.ObjectDisposedException: Cannot write to the response body, the response has completed. Object name: 'HttpResponseStream' or System.InvalidOperationException: Reading is already in progress..

I've created a minimal repo you can clone and try to reproduce the issue. It uses efcore with in memory database but the same happens with SQL Server. I've also added a readme for the few steps required to reproduce the issue.

Tried the same with an Api Controller and had no issues. NET 5, Microsoft.AspNetCore.OData 8.0.0-preview3, Visual Studio 2019 16.8.3

[portno]

Another misbehavior is that when GETing the same url with the same $expand twice at the same time, one request returns the result as expected and the other one the result but not expanded. Maybe this will not show up in the reproduction repo since there are no data in there.

[Inverness]

This seems like it is responsible for the various issues I'm having with response compression when trying this preview:

2021-01-05 12:31:24.8339 [13] #1 ERROR Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware - An unhandled exception has occurred while executing the request.
System.InvalidOperationException: Only one asynchronous reader or writer is allowed time at one time.
   at System.IO.Compression.DeflateStream.ThrowInvalidBeginCall()
   at System.IO.Compression.DeflateStream.WriteAsyncMemory(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.IO.Compression.DeflateStream.WriteAsync(Byte[] array, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.IO.Compression.GZipStream.WriteAsync(Byte[] array, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionBody.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.OData.Formatter.StreamWrapper.Write(Byte[] buffer, Int32 offset, Int32 count)
   at Microsoft.OData.MessageStreamWrapper.MessageStreamWrappingStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Xml.XmlUtf8RawTextWriter.FlushBuffer()
   at System.Xml.XmlUtf8RawTextWriter.Flush()
   at System.Xml.XmlWellFormedWriter.Flush()
   at Microsoft.OData.ODataMetadataOutputContext.WriteMetadataDocument()
   at Microsoft.OData.ODataMessageWriter.<>c.<WriteMetadataDocument>b__70_0(ODataOutputContext context)
   at Microsoft.OData.ODataMessageWriter.WriteToOutput(ODataPayloadKind payloadKind, Action`1 writeAction)
   at Microsoft.OData.ODataMessageWriter.WriteMetadataDocument()
   at Microsoft.AspNetCore.OData.Formatter.Serialization.ODataMetadataSerializer.<>c__DisplayClass1_0.<WriteObjectAsync>b__0()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__277_0(Object obj)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.OData.Formatter.ODataOutputFormatterHelper.WriteToStreamAsync(Type type, Object value, IEdmModel model, ODataVersion version, Uri baseAddress, MediaTypeHeaderValue contentType, HttpRequest request, IHeaderDictionary requestHeaders, ODataSerializerProvider serializerProvider)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultFilters>g__Awaited|27_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

And others:

2021-01-05 12:31:24.6169 [8] #1 ERROR Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware - An unhandled exception has occurred while executing the request.
System.InvalidOperationException: Only one asynchronous reader or writer is allowed time at one time.
   at System.IO.Compression.DeflateStream.ThrowInvalidBeginCall()
   at System.IO.Compression.DeflateStream.WriteAsyncMemory(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.IO.Compression.DeflateStream.WriteAsync(Byte[] array, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.IO.Compression.GZipStream.WriteAsync(Byte[] array, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionBody.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
2021-01-05 12:31:24.6169 [7] #1 ERROR Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware - An unhandled exception has occurred while executing the request.
System.InvalidOperationException: Only one asynchronous reader or writer is allowed time at one time.
   at System.IO.Compression.DeflateStream.ThrowInvalidBeginCall()
   at System.IO.Compression.DeflateStream.WriteAsyncMemory(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.IO.Compression.DeflateStream.WriteAsync(Byte[] array, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.IO.Compression.GZipStream.WriteAsync(Byte[] array, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionBody.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
2021-01-05 12:31:24.6169 [10] #1 ERROR Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware - An unhandled exception has occurred while executing the request.
System.InvalidOperationException: Only one asynchronous reader or writer is allowed time at one time.
   at System.IO.Compression.DeflateStream.ThrowInvalidBeginCall()
   at System.IO.Compression.DeflateStream.WriteAsyncMemory(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.IO.Compression.DeflateStream.WriteAsync(Byte[] array, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.IO.Compression.GZipStream.WriteAsync(Byte[] array, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionBody.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
2021-01-05 12:31:24.6658 [7] #2 WARN Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware - The response has already started, the error page middleware will not be executed.
2021-01-05 12:31:24.7588 [7] #13 ERROR Microsoft.AspNetCore.Server.Kestrel - Connection id "0HM5HL0BCH230", Request id "0HM5HL0BCH230:00000002": An unhandled exception was thrown by the application.
System.InvalidOperationException: Only one asynchronous reader or writer is allowed time at one time.
   at System.IO.Compression.DeflateStream.ThrowInvalidBeginCall()
   at System.IO.Compression.DeflateStream.WriteAsyncMemory(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
   at System.IO.Compression.DeflateStream.WriteAsync(Byte[] array, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at System.IO.Compression.GZipStream.WriteAsync(Byte[] array, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionBody.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)

[houbi56]

There's a concurrency issue when using the EnableQueryAttribute since 7.5.4. See https://github.com/OData/WebApi/issues/2390 It might be the culprit here as well.

[JanKotschenreuther]

Got the same issue, when a lot of request arrive at the same time, there is a chance that some of them end up in a System.ObjectDisposedException. Here you got the exception as JSON with StackTrace, maybe that helps solving:

{
    "ClassName": "System.ObjectDisposedException",
    "Message": "Cannot write to the response body, the response has completed.",
    "Data": null,
    "InnerException": null,
    "HelpURL": null,
    "StackTraceString": "   
         at Microsoft.AspNetCore.Server.IIS.Core.HttpResponseStream.ValidateState(CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.IIS.Core.HttpResponseStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.Server.IIS.Core.WrappingStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
         at Microsoft.AspNetCore.OData.Formatter.StreamWrapper.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
         at Microsoft.OData.MessageStreamWrapper.MessageStreamWrappingStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
         at Microsoft.OData.AsyncBufferedStream.FlushBuffersAsync(Queue`1 buffers)+MoveNext()
         at Microsoft.OData.TaskUtils.<>c__DisplayClass25_0.<Iterate>b__1(Task antecedent)
    ",
    "RemoteStackTraceString": null,
    "RemoteStackIndex": 0,
    "ExceptionMethod": null,
    "HResult": -2146232798,
    "Source": "Microsoft.AspNetCore.Server.IIS",
    "WatsonBuckets": null,
    "ObjectName": "HttpResponseStream"
}

[manureini]

I'm facing the same issue in version 8.0.10 Does someone have news about this issue or workarounds etc?

[Airex]

8.0.2 During heavy load there are cases when response contains data from other responses, even if response from other table

[julealgon]

@xuzhg this looks like the most critical issue currently tracked in the repo... is anyone actively looking into it? Could it be bumped in priority?

[anguis-datura]

It's critical. Please have a look with high priority.

[pogliad]

We got same issue. Waiting for fix. Thank you.

[manureini]

@xuzhg Is there any chance that you can look after this issue?

I can reproduce it very easily on my machine with version 8.0.12. To create some load I'm using these few lines: https://gist.github.com/manureini/71b0c6384b561179f68bcb28d5781b43

Everything will run without any exception.

When Adding await Task.Delay(1000); to WriteObjectAsync of a custom ODataResourceSetSerializer the exception will be thrown in the code of the gist. The result starts like this in my case: {"@odata.context":"http://localhost:5000/api/$metadata#Department(Person(),ApplicationShiftAssignments(Shift(Job())))" The context contains filter values of the other request.

Maybe my tests can help a little bit.