search

dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.
https://asp.net
MIT License
35.3k stars 9.97k forks source link

Block is backed by disposed slab #24557

Open davidfowl opened 4 years ago

davidfowl commented 4 years ago

https://dev.azure.com/dnceng/public/_build/results?buildId=756151&view=ms.vss-test-web.build-test-results-tab&runId=23472678&paneView=debug&resultId=121832

System.AggregateException : Exceptions occurred while accessing blocks (Block is backed by disposed slab)
---- System.InvalidOperationException : Block is backed by disposed slab

  at System.Buffers.DiagnosticMemoryPool.WhenAllBlocksReturnedAsync(TimeSpan timeout) in /_/src/Shared/Buffers.MemoryPool/DiagnosticMemoryPool.cs:line 162
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.MaxRequestBufferSizeTests.ServerShutsDownGracefullyWhenMaxRequestBufferSizeExceeded() in /_/src/Servers/Kestrel/test/FunctionalTests/MaxRequestBufferSizeTests.cs:line 281
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at System.Buffers.MemoryPoolThrowHelper.ThrowInvalidOperationException_BlockIsBackedByDisposedSlab(DiagnosticPoolBlock block) in /_/src/Shared/Buffers.MemoryPool/MemoryPoolThrowHelper.cs:line 57
   at System.Buffers.DiagnosticPoolBlock.GetSpan() in /_/src/Shared/Buffers.MemoryPool/DiagnosticPoolBlock.cs:line 112
   at System.Buffers.BuffersExtensions.CopyToMultiSegment[T](ReadOnlySequence`1& sequence, Span`1 destination)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 destination, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.MaxRequestBufferSizeTests.<>c__DisplayClass6_0.<<StartWebHost>b__3>d.MoveNext() in /_/src/Servers/Kestrel/test/FunctionalTests/MaxRequestBufferSizeTests.cs:line 330
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequest[TContext](IHttpApplication`1 application) in /_/src/Servers/Kestrel/Core/src/Internal/Http/HttpProtocol.cs:line 659

cc @halter73

halter73 commented 4 years ago

The timeout on this test is pretty tight. It's not surprising that it fails this way rarely, since the shutdown is meant to be abortive, the app func read loop only has a second to observe the request/connection being aborted and stop reading or else it will get this error.

I think our two options are to:

  1. Stop using the DiagnosticMemoryPoolFactory in this test allowLateReturn or otherwise.
  2. Make the time we wait for app funcs to complete after aborting connections configurable. See here.
ghost commented 4 years ago

Thanks for contacting us. We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

ryanbrandenburg commented 4 years ago

https://dev.azure.com/dnceng/public/_build/results?buildId=773979&view=ms.vss-test-web.build-test-results-tab&runId=24182790&resultId=122372&paneView=debug this error seems to be continuing.

BrennanConroy commented 3 years ago

Triage: We want to make the abort timeout internally configurable.

davidfowl commented 3 years ago

Can close this now. We don't have anymore slabs!

halter73 commented 2 weeks ago

We don't have anymore slabs, but the test still fails with an InvalidOperationException complaining that the "Block is backed by disposed slab". While the test-only exception has a misleading message, the block has still been disposed when we try to read the request body out of it in the test during an abortive shutdown.

I think our options are still either to:

  1. Stop using the DiagnosticMemoryPoolFactory in this test allowLateReturn or otherwise.
  2. Make the time we wait for app funcs to complete after aborting connections configurable. See here.

I see we wanted to make the hard-coded 1 second abort timeout configurable for this test, but I'm not sure it's worth the effort.

I think this test has always been flaky, but only became unquarantined shortly before this issue was filed. That said, the fact it hasn't been requarantined in the last 4 years must mean it's not that flaky.

Failing Test(s)

Error Message

System.AggregateException : Exceptions occurred while accessing blocks (Block is backed by disposed slab)
---- System.InvalidOperationException : Block is backed by disposed slab

Stacktrace

```text at System.Buffers.DiagnosticMemoryPool.WhenAllBlocksReturnedAsync(TimeSpan timeout) in /_/src/Shared/Buffers.MemoryPool/DiagnosticMemoryPool.cs:line 161 at Microsoft.AspNetCore.Server.Kestrel.Sockets.FunctionalTests.MaxRequestBufferSizeTests.ServerShutsDownGracefullyWhenMaxRequestBufferSizeExceeded() in /_/src/Servers/Kestrel/test/FunctionalTests/MaxRequestBufferSizeTests.cs:line 297 --- End of stack trace from previous location --- ----- Inner Stack Trace ----- at System.Buffers.MemoryPoolThrowHelper.ThrowInvalidOperationException_BlockIsBackedByDisposedSlab(DiagnosticPoolBlock block) in /_/src/Shared/Buffers.MemoryPool/MemoryPoolThrowHelper.cs:line 57 at System.Buffers.DiagnosticPoolBlock.GetSpan() in /_/src/Shared/Buffers.MemoryPool/DiagnosticPoolBlock.cs:line 112 at System.Buffers.BuffersExtensions.CopyToMultiSegment[T](ReadOnlySequence`1& sequence, Span`1 destination) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 destination, CancellationToken cancellationToken) at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Threading.Tasks.ValueTask`1.GetTaskForValueTaskSource(IValueTaskSource`1 t) --- End of stack trace from previous location --- at Microsoft.AspNetCore.Server.Kestrel.Sockets.FunctionalTests.MaxRequestBufferSizeTests.<>c__DisplayClass6_0.<b__4>d.MoveNext() in /_/src/Servers/Kestrel/test/FunctionalTests/MaxRequestBufferSizeTests.cs:line 348 --- End of stack trace from previous location --- at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application) in /_/src/Servers/Kestrel/Core/src/Internal/Http/HttpProtocol.cs:line 683 ```

Logs

```text | [0.001s] TestLifetime Information: Starting test ServerShutsDownGracefullyWhenMaxRequestBufferSizeExceeded at 2024-09-17T20:22:25 | [0.003s] Microsoft.Extensions.Hosting.Internal.Host Debug: Hosting starting | [0.004s] Microsoft.Hosting.Lifetime Information: Now listening on: http://127.0.0.1:50432 | [0.004s] Microsoft.AspNetCore.Hosting.Diagnostics Debug: Loaded hosting startup assembly Sockets.FunctionalTests | [0.004s] Microsoft.Hosting.Lifetime Information: Application started. Press Ctrl+C to shut down. | [0.007s] Microsoft.Hosting.Lifetime Information: Hosting environment: Production | [0.007s] Microsoft.Hosting.Lifetime Information: Content root path: C:\h\w\B4C50A24\w\9FB608D1\e | [0.007s] Microsoft.Extensions.Hosting.Internal.Host Debug: Hosting started | [0.007s] Microsoft.AspNetCore.Server.Kestrel.Connections Debug: Connection id "0HN6NAPIMSSUK" accepted. | [0.007s] Microsoft.AspNetCore.Server.Kestrel.Connections Debug: Connection id "0HN6NAPIMSSUK" started. | [0.008s] Microsoft.AspNetCore.Hosting.Diagnostics Information: Request starting HTTP/1.1 POST http:/// - - 104857600 | [0.008s] Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets Debug: Connection id "0HN6NAPIMSSUK" paused. | [0.123s] Microsoft.Extensions.Hosting.Internal.Host Debug: Hosting stopping | [0.123s] Microsoft.Hosting.Lifetime Information: Application is shutting down... | [30.976s] Microsoft.AspNetCore.Server.Kestrel.Connections Debug: Some connections failed to close gracefully during server shutdown. | [30.976s] Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets Debug: Connection id "0HN6NAPIMSSUK" sending RST because: "The connection was aborted because the server is shutting down and request processing didn't complete within the time specified by HostOptions.ShutdownTimeout." | [33.322s] Microsoft.AspNetCore.Server.Kestrel.Connections Debug: Some connections failed to abort during server shutdown. | [33.322s] Microsoft.Extensions.Hosting.Internal.Host Debug: Hosting stopped | [33.323s] Microsoft.AspNetCore.Server.Kestrel Debug: Connection id "0HN6NAPIMSSUK", Request id "0HN6NAPIMSSUK:00000001": started reading request body. | [33.324s] Microsoft.AspNetCore.Server.Kestrel Error: Connection id "0HN6NAPIMSSUK", Request id "0HN6NAPIMSSUK:00000001": An unhandled exception was thrown by the application. | System.InvalidOperationException: Block is backed by disposed slab | at System.Buffers.MemoryPoolThrowHelper.ThrowInvalidOperationException_BlockIsBackedByDisposedSlab(DiagnosticPoolBlock block) in /_/src/Shared/Buffers.MemoryPool/MemoryPoolThrowHelper.cs:line 57 | at System.Buffers.DiagnosticPoolBlock.GetSpan() in /_/src/Shared/Buffers.MemoryPool/DiagnosticPoolBlock.cs:line 112 | at System.Buffers.BuffersExtensions.CopyToMultiSegment[T](ReadOnlySequence`1& sequence, Span`1 destination) | at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 destination, CancellationToken cancellationToken) | at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) | at System.Threading.Tasks.ValueTask`1.GetTaskForValueTaskSource(IValueTaskSource`1 t) | --- End of stack trace from previous location --- | at Microsoft.AspNetCore.Server.Kestrel.Sockets.FunctionalTests.MaxRequestBufferSizeTests.<>c__DisplayClass6_0.<b__4>d.MoveNext() in /_/src/Servers/Kestrel/test/FunctionalTests/MaxRequestBufferSizeTests.cs:line 348 | --- End of stack trace from previous location --- | at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application) in /_/src/Servers/Kestrel/Core/src/Internal/Http/HttpProtocol.cs:line 683 | [33.325s] Microsoft.AspNetCore.Server.Kestrel Warning: Connection processing ended abnormally. | System.ObjectDisposedException: Cannot access a disposed object. | Object name: 'MemoryPool'. | at System.Buffers.MemoryPoolThrowHelper.ThrowObjectDisposedException(ExceptionArgument argument) in /_/src/Shared/Buffers.MemoryPool/MemoryPoolThrowHelper.cs:line 94 | at System.Buffers.DiagnosticMemoryPool.Rent(Int32 size) in /_/src/Shared/Buffers.MemoryPool/DiagnosticMemoryPool.cs:line 56 | at System.IO.Pipelines.Pipe.RentMemory(BufferSegment segment, Int32 sizeHint) | at System.IO.Pipelines.Pipe.AllocateWriteHeadSynchronized(Int32 sizeHint) | at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1OutputProducer.WriteResponseHeaders(Int32 statusCode, String reasonPhrase, HttpResponseHeaders responseHeaders, Boolean autoChunk, Boolean appComplete) in /_/src/Servers/Kestrel/Core/src/Internal/Http/Http1OutputProducer.cs:line 348 | at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProduceStart(Boolean appCompleted) in /_/src/Servers/Kestrel/Core/src/Internal/Http/HttpProtocol.cs:line 1002 | at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProduceEnd() in /_/src/Servers/Kestrel/Core/src/Internal/Http/HttpProtocol.cs:line 1062 | at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application) in /_/src/Servers/Kestrel/Core/src/Internal/Http/HttpProtocol.cs:line 753 | at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application) in /_/src/Servers/Kestrel/Core/src/Internal/Http/HttpProtocol.cs:line 583 | [33.325s] Microsoft.AspNetCore.Server.Kestrel.Connections Debug: Connection id "0HN6NAPIMSSUK" disconnecting. | [33.325s] Microsoft.AspNetCore.Server.Kestrel.Connections Debug: Connection id "0HN6NAPIMSSUK" stopped. | [33.326s] Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets Debug: Connection id "0HN6NAPIMSSUK" resumed. | [33.327s] Microsoft.AspNetCore.Server.Kestrel.Sockets.FunctionalTests.MaxRequestBufferSizeTests Error: Test threw an exception. | System.AggregateException: Exceptions occurred while accessing blocks (Block is backed by disposed slab) | ---> System.InvalidOperationException: Block is backed by disposed slab | at System.Buffers.MemoryPoolThrowHelper.ThrowInvalidOperationException_BlockIsBackedByDisposedSlab(DiagnosticPoolBlock block) in /_/src/Shared/Buffers.MemoryPool/MemoryPoolThrowHelper.cs:line 57 | at System.Buffers.DiagnosticPoolBlock.GetSpan() in /_/src/Shared/Buffers.MemoryPool/DiagnosticPoolBlock.cs:line 112 | at System.Buffers.BuffersExtensions.CopyToMultiSegment[T](ReadOnlySequence`1& sequence, Span`1 destination) | at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 destination, CancellationToken cancellationToken) | at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) | at System.Threading.Tasks.ValueTask`1.GetTaskForValueTaskSource(IValueTaskSource`1 t) | --- End of stack trace from previous location --- | at Microsoft.AspNetCore.Server.Kestrel.Sockets.FunctionalTests.MaxRequestBufferSizeTests.<>c__DisplayClass6_0.<b__4>d.MoveNext() in /_/src/Servers/Kestrel/test/FunctionalTests/MaxRequestBufferSizeTests.cs:line 348 | --- End of stack trace from previous location --- | at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application) in /_/src/Servers/Kestrel/Core/src/Internal/Http/HttpProtocol.cs:line 683 | --- End of inner exception stack trace --- | at System.Buffers.DiagnosticMemoryPool.WhenAllBlocksReturnedAsync(TimeSpan timeout) in /_/src/Shared/Buffers.MemoryPool/DiagnosticMemoryPool.cs:line 161 | at Microsoft.AspNetCore.Server.Kestrel.Sockets.FunctionalTests.MaxRequestBufferSizeTests.ServerShutsDownGracefullyWhenMaxRequestBufferSizeExceeded() in /_/src/Servers/Kestrel/test/FunctionalTests/MaxRequestBufferSizeTests.cs:line 297 | at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass46_0.<b__1>d.MoveNext() in /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs:line 253 | --- End of stack trace from previous location --- | at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 48 | at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in /_/src/xunit.core/Sdk/ExceptionAggregator.cs:line 90 | [33.328s] TestLifetime Information: Finished test ServerShutsDownGracefullyWhenMaxRequestBufferSizeExceeded in 33.3268565s ```

Build

JamesNK commented 2 weeks ago

I fixed an error like this here: https://github.com/dotnet/aspnetcore/pull/56281