dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.47k stars 4.76k forks source link

System.Net.Quic is leaking UnobservedTaskExceptions #80111

Closed MihaZupan closed 4 months ago

MihaZupan commented 1 year ago

I have a console app with HttpClient that continuously sends requests to a localhost Kestrel instance. If I randomly shut down and restart the server while HttpClient is making requests, I can see sporadic UnobservedTaskExceptions coming from System.Net.Quic.

System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Operation aborted.)
 ---> System.Net.Quic.QuicException: Operation aborted.
   at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\Internal\ResettableValueTaskSource.cs:line 157
   at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception exception, Boolean final) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\Internal\ResettableValueTaskSource.cs:line 221
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicStream.cs:line 634
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicStream.cs:line 671
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicStream.cs:line 695
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\Internal\ResettableValueTaskSource.cs:line 235
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicStream.cs:line 306
   at System.Net.Http.Http3Connection.<>c__DisplayClass45_0.<<ProcessServerControlStreamAsync>g__ReadFrameEnvelopeAsync|0>d.MoveNext() in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs:line 679
--- End of stack trace from previous location ---
   at System.Net.Http.Http3Connection.ProcessServerControlStreamAsync(QuicStream stream, ArrayBuffer buffer) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs:line 624
   at System.Net.Http.Http3Connection.ProcessServerStreamAsync(QuicStream stream) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs:line 520
   at System.Net.Http.Http3Connection.ProcessServerStreamAsync(QuicStream stream) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs:line 575
   --- End of inner exception stack trace ---
System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Operation aborted.)
 ---> System.Net.Quic.QuicException: Operation aborted.
   at System.Net.Quic.QuicConnection.HandleEventShutdownComplete() in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 492
   at System.Net.Quic.QuicConnection.HandleConnectionEvent(QUIC_CONNECTION_EVENT& connectionEvent) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 571
   at System.Net.Quic.QuicConnection.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_CONNECTION_EVENT* connectionEvent) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 598
--- End of stack trace from previous location ---
   at System.Net.Quic.QuicConnection.AcceptInboundStreamAsync(CancellationToken cancellationToken) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 402
   at System.Net.Http.Http3Connection.AcceptStreamsAsync() in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs:line 438
   --- End of inner exception stack trace ---
System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (The server is currently unreachable.)
 ---> System.Net.Quic.QuicException: The server is currently unreachable.
   at System.Net.Quic.QuicConnection.HandleEventShutdownInitiatedByTransport(_SHUTDOWN_INITIATED_BY_TRANSPORT_e__Struct& data) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 470
   at System.Net.Quic.QuicConnection.HandleConnectionEvent(QUIC_CONNECTION_EVENT& connectionEvent) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 569
   at System.Net.Quic.QuicConnection.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_CONNECTION_EVENT* connectionEvent) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 598
--- End of stack trace from previous location ---
   at System.Net.Quic.ValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\Internal\ValueTaskSource.cs:line 176
   at System.Net.Quic.QuicConnection.FinishConnectAsync(QuicClientConnectionOptions options, CancellationToken cancellationToken) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 312
   at System.Net.Quic.QuicConnection.ConnectAsync(QuicClientConnectionOptions options, CancellationToken cancellationToken) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 68
   at System.Net.Quic.QuicConnection.ConnectAsync(QuicClientConnectionOptions options, CancellationToken cancellationToken) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 73
   at System.Net.Http.ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, DnsEndPoint endPoint, TimeSpan idleTimeout, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\ConnectHelper.cs:line 120
   --- End of inner exception stack trace ---
ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Issue Details
I have a console app with HttpClient that continuously sends requests to a localhost Kestrel instance. If I randomly shut down and restart the server while HttpClient is making requests, I can see sporadic `UnobservedTaskException`s coming from `System.Net.Quic`. ``` System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Operation aborted.) ---> System.Net.Quic.QuicException: Operation aborted. at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception exception, Boolean final) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\Internal\ResettableValueTaskSource.cs:line 157 at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception exception, Boolean final) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\Internal\ResettableValueTaskSource.cs:line 221 at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicStream.cs:line 634 at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicStream.cs:line 671 at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_STREAM_EVENT* streamEvent) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicStream.cs:line 695 --- End of stack trace from previous location --- at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\Internal\ResettableValueTaskSource.cs:line 235 at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicStream.cs:line 306 at System.Net.Http.Http3Connection.<>c__DisplayClass45_0.<g__ReadFrameEnvelopeAsync|0>d.MoveNext() in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs:line 679 --- End of stack trace from previous location --- at System.Net.Http.Http3Connection.ProcessServerControlStreamAsync(QuicStream stream, ArrayBuffer buffer) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs:line 624 at System.Net.Http.Http3Connection.ProcessServerStreamAsync(QuicStream stream) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs:line 520 at System.Net.Http.Http3Connection.ProcessServerStreamAsync(QuicStream stream) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs:line 575 --- End of inner exception stack trace --- ``` ``` System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Operation aborted.) ---> System.Net.Quic.QuicException: Operation aborted. at System.Net.Quic.QuicConnection.HandleEventShutdownComplete() in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 492 at System.Net.Quic.QuicConnection.HandleConnectionEvent(QUIC_CONNECTION_EVENT& connectionEvent) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 571 at System.Net.Quic.QuicConnection.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_CONNECTION_EVENT* connectionEvent) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 598 --- End of stack trace from previous location --- at System.Net.Quic.QuicConnection.AcceptInboundStreamAsync(CancellationToken cancellationToken) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 402 at System.Net.Http.Http3Connection.AcceptStreamsAsync() in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs:line 438 --- End of inner exception stack trace --- ``` ``` System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (The server is currently unreachable.) ---> System.Net.Quic.QuicException: The server is currently unreachable. at System.Net.Quic.QuicConnection.HandleEventShutdownInitiatedByTransport(_SHUTDOWN_INITIATED_BY_TRANSPORT_e__Struct& data) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 470 at System.Net.Quic.QuicConnection.HandleConnectionEvent(QUIC_CONNECTION_EVENT& connectionEvent) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 569 at System.Net.Quic.QuicConnection.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_CONNECTION_EVENT* connectionEvent) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 598 --- End of stack trace from previous location --- at System.Net.Quic.ValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\Internal\ValueTaskSource.cs:line 176 at System.Net.Quic.QuicConnection.FinishConnectAsync(QuicClientConnectionOptions options, CancellationToken cancellationToken) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 312 at System.Net.Quic.QuicConnection.ConnectAsync(QuicClientConnectionOptions options, CancellationToken cancellationToken) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 68 at System.Net.Quic.QuicConnection.ConnectAsync(QuicClientConnectionOptions options, CancellationToken cancellationToken) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 73 at System.Net.Http.ConnectHelper.ConnectQuicAsync(HttpRequestMessage request, DnsEndPoint endPoint, TimeSpan idleTimeout, SslClientAuthenticationOptions clientAuthenticationOptions, CancellationToken cancellationToken) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\ConnectHelper.cs:line 120 --- End of inner exception stack trace --- ```
Author: MihaZupan
Assignees: -
Labels: `bug`, `area-System.Net.Quic`
Milestone: -
JustArchi commented 1 year ago

I can confirm this with entirely different project.

``` 2023-01-12 02:50:08|ArchiSteamFarm-2292|FATAL|ASF|OnUnobservedTaskException() System.Net.Quic.QuicException: Reading has been aborted on the stream. at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception, Boolean) at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception, Boolean ) at System.Net.Quic.QuicStream.Abort(QuicAbortDirection, Int64) at System.Net.Http.Http3RequestStream.AbortStream() at System.Net.Http.Http3RequestStream.Dispose() at System.Net.Http.Http3RequestStream.Http3ReadStream.Dispose(Boolean) at System.IO.Stream.Close() at System.IO.Compression.BrotliStream.Dispose(Boolean) at System.IO.Stream.Close() at System.IO.StreamReader.Dispose(Boolean) at System.IO.StreamReader.Close() at Newtonsoft.Json.JsonTextReader.Close() at Newtonsoft.Json.JsonReader.Dispose(Boolean disposing) at Newtonsoft.Json.JsonReader.System.IAsyncDisposable.DisposeAsync() at System.Runtime.CompilerServices.ConfiguredAsyncDisposable.DisposeAsync() at ArchiSteamFarm.Web.WebBrowser.UrlGetToJsonObject[T](Uri request, IReadOnlyCollection`1 headers, Uri referer, ERequestOptions requestOptions, Byte maxTries, Int32 rateLimitingDelay) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object) at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) at System.Threading.Tasks.Task.RunContinuations(Object) at System.Threading.Tasks.Task`1.TrySetResult(TResult) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult) at ArchiSteamFarm.Web.WebBrowser.UrlGetToStream(Uri request, IReadOnlyCollection`1 headers, Uri referer, ERequestOptions requestOptions, Byte maxTries, Int32 rateLimitingDelay) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object) at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) at System.Threading.Tasks.Task.RunContinuations(Object) at System.Threading.Tasks.Task`1.TrySetResult(TResult) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult) at ArchiSteamFarm.Web.WebBrowser.InternalGet(Uri request, IReadOnlyCollection`1 headers, Uri referer, ERequestOptions requestOptions, HttpCompletionOption httpCompletionOption) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object) at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) at System.Threading.Tasks.Task.RunContinuations(Object) at System.Threading.Tasks.Task`1.TrySetResult(TResult) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult) at ArchiSteamFarm.Web.WebBrowser.InternalRequest[T](Uri request, HttpMethod httpMethod, IReadOnlyCollection`1 headers, T data, Uri referer, ERequestOptions requestOptions, HttpCompletionOption httpCompletionOption, Byte maxRedirections) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object) at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) at System.Threading.Tasks.Task.RunContinuations(Object) at System.Threading.Tasks.Task`1.TrySetResult(TResult) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult) at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage, HttpCompletionOption, CancellationTokenSource, Boolean, CancellationTokenSource, CancellationToken) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object) at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) at System.Threading.Tasks.Task.RunContinuations(Object) at System.Threading.Tasks.Task`1.TrySetResult(TResult) at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult) at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage, Boolean, CancellationToken) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object) at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) at System.Threading.Tasks.Task.RunContinuations(Object) at System.Threading.Tasks.Task`1.TrySetResult(TResult) at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage, Boolean, Boolean, CancellationToken) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object) at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) at System.Threading.Tasks.Task.RunContinuations(Object) at System.Threading.Tasks.Task`1.TrySetResult(TResult) at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult) at System.Net.Http.HttpConnectionPool.TrySendUsingHttp3Async(HttpRequestMessage, CancellationToken) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object) at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) at System.Threading.Tasks.Task.RunContinuations(Object) at System.Threading.Tasks.Task`1.TrySetResult(TResult) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult) at System.Net.Http.Http3Connection.SendAsync(HttpRequestMessage, Int64, CancellationToken) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object) at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) at System.Threading.Tasks.Task.RunContinuations(Object) at System.Threading.Tasks.Task`1.TrySetResult(TResult) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult) at System.Net.Http.Http3RequestStream.SendAsync(CancellationToken) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object) at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) at System.Threading.Tasks.Task.RunContinuations(Object) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult() at System.Net.Http.Http3RequestStream.ReadResponseAsync(CancellationToken) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object) at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) at System.Threading.Tasks.Task.RunContinuations(Object) at System.Threading.Tasks.Task`1.TrySetResult(TResult) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1, TResult) at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult) at System.Net.Http.Http3RequestStream.ReadFrameEnvelopeAsync(CancellationToken) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object) at System.Threading.ExecutionContext.RunInternal(ExecutionContext, ContextCallback, Object) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext() at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox, Boolean) at System.Threading.Tasks.Task.RunContinuations(Object) at System.Threading.Tasks.Task`1.TrySetResult(TResult) at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder`1.SetResult(TResult) at System.Net.Quic.QuicStream.ReadAsync(Memory`1, CancellationToken ) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object) at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread, ExecutionContext, ContextCallback, Object) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread) at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecuteFromThreadPool(Thread) at System.Threading.ThreadPoolWorkQueue.Dispatch() at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() at System.Threading.Thread.StartCallback() --- End of stack trace from previous location --- ``` ``` 2023-01-12 03:05:01|ArchiSteamFarm-2292|FATAL|ASF|OnUnobservedTaskException() System.Net.Quic.QuicException: The connection timed out from inactivity. at System.Net.Quic.QuicConnection.HandleEventShutdownInitiatedByTransport(_SHUTDOWN_INITIATED_BY_TRANSPORT_e__Struct&) at System.Net.Quic.QuicConnection.HandleConnectionEvent(QUIC_CONNECTION_EVENT&) at System.Net.Quic.QuicConnection.NativeCallback(QUIC_HANDLE*, Void*, QUIC_CONNECTION_EVENT*) --- End of stack trace from previous location --- at System.Net.Quic.QuicConnection.AcceptInboundStreamAsync(CancellationToken ) at System.Net.Http.Http3Connection.AcceptStreamsAsync() 2023-01-12 03:05:01|ArchiSteamFarm-2292|FATAL|ASF|OnUnobservedTaskException() System.Net.Quic.QuicException: The connection timed out from inactivity. at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception, Boolean) at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception, Boolean ) at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct&) at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT&) at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE*, Void*, QUIC_STREAM_EVENT*) --- End of stack trace from previous location --- at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16) at System.Net.Quic.QuicStream.ReadAsync(Memory`1, CancellationToken ) at System.IO.Stream.g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.Http3Connection.ProcessServerStreamAsync(QuicStream) at System.Net.Http.Http3Connection.ProcessServerStreamAsync(QuicStream) 2023-01-12 03:05:01|ArchiSteamFarm-2292|FATAL|ASF|OnUnobservedTaskException() System.Net.Quic.QuicException: The connection timed out from inactivity. at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception, Boolean) at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception, Boolean ) at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct&) at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT&) at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE*, Void*, QUIC_STREAM_EVENT*) --- End of stack trace from previous location --- at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16) at System.Net.Quic.QuicStream.ReadAsync(Memory`1, CancellationToken ) at System.IO.Stream.g__Core|27_0(Stream source, Stream destination, Int32 bufferSize, CancellationToken cancellationToken) at System.Net.Http.Http3Connection.ProcessServerStreamAsync(QuicStream) at System.Net.Http.Http3Connection.ProcessServerStreamAsync(QuicStream) 2023-01-12 03:05:01|ArchiSteamFarm-2292|FATAL|ASF|OnUnobservedTaskException() System.Net.Quic.QuicException: The connection timed out from inactivity. at System.Net.Quic.ResettableValueTaskSource.TryComplete(Exception, Boolean) at System.Net.Quic.ResettableValueTaskSource.TrySetException(Exception, Boolean ) at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct&) at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT&) at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE*, Void*, QUIC_STREAM_EVENT*) --- End of stack trace from previous location --- at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16) at System.Net.Quic.QuicStream.ReadAsync(Memory`1, CancellationToken ) at System.Net.Http.Http3Connection.<>c__DisplayClass45_0.<g__ReadFrameEnvelopeAsync|0>d.MoveNext() --- End of stack trace from previous location --- at System.Net.Http.Http3Connection.ProcessServerControlStreamAsync(QuicStream, ArrayBuffer) at System.Net.Http.Http3Connection.ProcessServerStreamAsync(QuicStream) at System.Net.Http.Http3Connection.ProcessServerStreamAsync(QuicStream) ```

The application seems to work fine however.

ManickaP commented 1 year ago

Triage: we should fix this in 8.0.

alexrp commented 1 year ago

Just wanted to confirm given https://github.com/dotnet/runtime/issues/80111#issuecomment-1405296246, was this moved to 9.0.0 intentionally?

ManickaP commented 1 year ago

Yes, it was moved. Unfortunately, we didn't have a chance to look into it. And as this doesn't affect application behavior, it got lower priority. It's annoying, but not detrimental.

liveans commented 8 months ago

Looks like this is already fixed by some changes, can't repro it on main. @MihaZupan can you repro this on main?

MihaZupan commented 8 months ago

I can consistently repro on 8, but wasn't able to on 9.0 anymore. Might have been fixed recently, closing. Feel free to reopen if anyone's still seeing it.

MihaZupan commented 5 months ago

Doing a simple HTTP test run

.dotnet/dotnet build .\src\libraries\System.Net.Http\tests\FunctionalTests\ /t:Test

while watching for unobserved exceptions

[Fact]
public async Task WatchForUnobserved()
{
    bool seenUnobservedExceptions = false;

    EventHandler<UnobservedTaskExceptionEventArgs> eventHandler = (_, e) =>
    {
        _output.WriteLine(e.Exception.ToString());
        seenUnobservedExceptions = true;
    };

    TaskScheduler.UnobservedTaskException += eventHandler;
    await Task.Delay(10_000);
    TaskScheduler.UnobservedTaskException -= eventHandler;

    Assert.False(seenUnobservedExceptions);
}

I'm still seeing 100s of such unobserved exceptions (mainly from Quic):

System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Cannot access a disposed object.
Object name: 'System.Net.Quic.QuicConnection'.)
---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Quic.QuicConnection'.
 at System.Net.Quic.QuicConnection.HandleEventShutdownComplete() in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 652
 at System.Net.Quic.QuicConnection.HandleConnectionEvent(QUIC_CONNECTION_EVENT& connectionEvent) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 738
 at System.Net.Quic.QuicConnection.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_CONNECTION_EVENT* connectionEvent) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 771
--- End of stack trace from previous location ---
System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Operation aborted.)
---> System.Net.Quic.QuicException: Operation aborted.
 at System.Net.Quic.QuicConnection.HandleEventShutdownComplete() in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 652
 at System.Net.Quic.QuicConnection.HandleConnectionEvent(QUIC_CONNECTION_EVENT& connectionEvent) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 738
 at System.Net.Quic.QuicConnection.NativeCallback(QUIC_HANDLE* connection, Void* context, QUIC_CONNECTION_EVENT* connectionEvent) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 771
--- End of stack trace from previous location ---
 at System.Net.Quic.QuicConnection.AcceptInboundStreamAsync(CancellationToken cancellationToken) in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Quic\src\System\Net\Quic\QuicConnection.cs:line 565
 at System.Net.Http.Http3Connection.AcceptStreamsAsync() in C:\MihaZupan\runtime-debug\src\libraries\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\Http3Connection.cs:line 523
 --- End of inner exception stack trace ---