dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.9k stars 4.63k forks source link

[H/3] Stress run in Debug fails on assert #93713

Open ManickaP opened 10 months ago

ManickaP commented 10 months ago

Easy to reproduce locally with: .\run-stress-debug-debug.ps1 -cancelRate 1 -http 3.0

We use duplex streaming with custom contents. So when the cancellation happens, response reading side gets cancelled and disposes Http3RequestStream which disposes the ArrayBuffer fields. But the writing side might not have reacted to the CancellationToken yet and the next thing it does is touching the ArrayBuffer: https://github.com/dotnet/runtime/blob/c5e7081f78aaae13238ed54c2c359999247e09fa/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs#L465

Nothing bad will happen as the next QuicStream.WriteAsync will kill this task.

Note that similar problem was encountered with these asserts: https://github.com/dotnet/runtime/blob/c5e7081f78aaae13238ed54c2c359999247e09fa/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs#L716-L718

In general, we do not access Http3RequestStream fields in a thread-safe manner in case of duplex streaming which can cause problems when we Dispose the object from the reading side while still having writing side active.

Question is whether this is worth fixing or not. Either way, we should have a paper trail about the decision.

Process terminated. Assertion failed.
Expected 2 <= 0
   at System.Net.ArrayBuffer.Discard(Int32 byteCount) in C:\Users\mapichov\runtime\src\libraries\Common\src\System\Net\ArrayBuffer.cs:line 103
   at System.Net.Http.Http3RequestStream.WriteRequestContentAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in C:\Users\mapichov\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs:line 484
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\ExecutionContext.cs:line 179
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Runtime\CompilerServices\AsyncTaskMethodBuilderT.cs:line 352
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\TaskContinuation.cs:line 795
   at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\Task.cs:line 3456
   at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder.SetResult() in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Runtime\CompilerServices\AsyncValueTaskMethodBuilder.cs:line 41
   at System.Net.Quic.QuicStream.WriteAsync(ReadOnlyMemory`1 buffer, Boolean completeWrites, CancellationToken cancellationToken) in C:\Users\mapichov\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicStream.cs:line 419
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\ExecutionContext.cs:line 264
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Runtime\CompilerServices\AsyncTaskMethodBuilderT.cs:line 352
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\ThreadPoolWorkQueue.cs:line 913
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\PortableThreadPool.WorkerThread.NonBrowser.cs:line 102
   at System.Threading.Thread.StartCallback() in C:\Users\mapichov\runtime\src\coreclr\System.Private.CoreLib\src\System\Threading\Thread.CoreCLR.cs:line 104
ghost commented 10 months ago

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Issue Details
Easy to reproduce locally with: `.\run-stress-debug-debug.ps1 -cancelRate 1 -http 3.0` We use duplex streaming with custom contents. So when the cancellation happens, response reading side gets cancelled and disposes `Http3RequestStream` which disposes the `ArrayBuffer` fields. But the writing side might not have reacted to the `CancellationToken` yet and the next thing it does is touching the `ArrayBuffer`: https://github.com/dotnet/runtime/blob/c5e7081f78aaae13238ed54c2c359999247e09fa/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/Http3RequestStream.cs#L465 Nothing bad will happen as the next `QuicStream.WriteAsync` will kill this task. Note that similar problem was encountered with these asserts: https://github.com/dotnet/runtime/blob/c5e7081f78aaae13238ed54c2c359999247e09fa/src/libraries/System.Net.Quic/src/System/Net/Quic/QuicStream.cs#L716-L718 In general, we do not access `Http3RequestStream` fields in a thread-safe manner in case of duplex streaming which can cause problems when we `Dispose` the object from the reading side while still having writing side active. **Question is whether this is worth fixing or not.** Either way, we should have a paper trail about the decision. ``` Process terminated. Assertion failed. Expected 2 <= 0 at System.Net.ArrayBuffer.Discard(Int32 byteCount) in C:\Users\mapichov\runtime\src\libraries\Common\src\System\Net\ArrayBuffer.cs:line 103 at System.Net.Http.Http3RequestStream.WriteRequestContentAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken) in C:\Users\mapichov\runtime\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3RequestStream.cs:line 484 at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\ExecutionContext.cs:line 179 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Runtime\CompilerServices\AsyncTaskMethodBuilderT.cs:line 352 at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\TaskContinuation.cs:line 795 at System.Threading.Tasks.Task.RunContinuations(Object continuationObject) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\Tasks\Task.cs:line 3456 at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder.SetResult() in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Runtime\CompilerServices\AsyncValueTaskMethodBuilder.cs:line 41 at System.Net.Quic.QuicStream.WriteAsync(ReadOnlyMemory`1 buffer, Boolean completeWrites, CancellationToken cancellationToken) in C:\Users\mapichov\runtime\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicStream.cs:line 419 at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\ExecutionContext.cs:line 264 at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread) in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Runtime\CompilerServices\AsyncTaskMethodBuilderT.cs:line 352 at System.Threading.ThreadPoolWorkQueue.Dispatch() in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\ThreadPoolWorkQueue.cs:line 913 at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in C:\Users\mapichov\runtime\src\libraries\System.Private.CoreLib\src\System\Threading\PortableThreadPool.WorkerThread.NonBrowser.cs:line 102 at System.Threading.Thread.StartCallback() in C:\Users\mapichov\runtime\src\coreclr\System.Private.CoreLib\src\System\Threading\Thread.CoreCLR.cs:line 104 ```
Author: ManickaP
Assignees: -
Labels: `area-System.Net.Http`
Milestone: -