Open jkotas opened 4 years ago
Hit again in #33922
Hit again in #33933
Another in #33820.
Broken since https://github.com/dotnet/runtime/pull/33158
Found a failure with a similar stacktrace on test
System.Net.Http.Functional.Tests.PlatformHandlerTest_Cookies_Http2.GetAsyncWithRedirect_ReceiveSetCookie_CookieSent [FAIL]
System.Net.Http.Functional.Tests.PlatformHandlerTest_Cookies_Http2.GetAsyncWithRedirect_ReceiveSetCookie_CookieSent [FAIL]
System.Net.Http.HttpRequestException : Error while copying content to a stream.
---- System.IO.IOException : The read operation failed, see inner exception.
-------- System.Net.Http.WinHttpException : Error 12030 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'The connection with the server was terminated abnormally'.
Stack Trace:
/_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs(446,0): at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
/_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(551,0): at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, CancellationToken callerToken, Int64 timeoutTime)
/_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cookies.cs(502,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Cookies.<GetAsyncWithRedirect_ReceiveSetCookie_CookieSent>b__23_0(Uri url)
/_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(83,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
/_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(111,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
/_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs(32,0): at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass4_0.<<CreateClientAndServerAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
/_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(38,0): at System.Threading.Tasks.TaskTimeoutExtensions.TimeoutAfter(Task task, TimeSpan timeout)
/_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs(243,0): at System.Net.Test.Common.Http2LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options)
/_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(38,0): at System.Threading.Tasks.TaskTimeoutExtensions.TimeoutAfter(Task task, TimeSpan timeout)
/_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Cookies.cs(493,0): at System.Net.Http.Functional.Tests.HttpClientHandlerTest_Cookies.GetAsyncWithRedirect_ReceiveSetCookie_CookieSent()
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
/_/src/libraries/Common/src/System/Threading/Tasks/RendezvousAwaitable.cs(64,0): at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
/_/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpResponseStream.cs(127,0): at System.Net.Http.WinHttpResponseStream.CopyToAsyncCore(Stream destination, Byte[] buffer, CancellationToken cancellationToken)
/_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs(440,0): at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
----- Inner Stack Trace -----
/_/src/coreclr/src/System.Private.CoreLib/src/System/Environment.CoreCLR.cs(95,0): at System.Environment.get_StackTrace()
/_/src/libraries/Common/src/System/Runtime/ExceptionServices/ExceptionStackTrace.cs(24,0): at System.Runtime.ExceptionServices.ExceptionStackTrace.AddCurrentStack(Exception exception)
/_/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpException.cs(59,0): at System.Net.Http.WinHttpException.CreateExceptionUsingError(Int32 error, String nameOfCalledFunction)
/_/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpRequestCallback.cs(333,0): at System.Net.Http.WinHttpRequestCallback.OnRequestError(WinHttpRequestState state, WINHTTP_ASYNC_RESULT asyncResult)
/_/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpRequestCallback.cs(104,0): at System.Net.Http.WinHttpRequestCallback.RequestCallback(IntPtr handle, WinHttpRequestState state, UInt32 internetStatus, IntPtr statusInformation, UInt32 statusInformationLength)
/_/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpRequestCallback.cs(47,0): at System.Net.Http.WinHttpRequestCallback.WinHttpCallback(IntPtr handle, IntPtr context, UInt32 internetStatus, IntPtr statusInformation, UInt32 statusInformationLength)
at Interop.WinHttp.WinHttpQueryDataAvailable(SafeWinHttpHandle requestHandle, IntPtr parameterIgnoredAndShouldBeNullForAsync)
/_/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpResponseStream.cs(122,0): at System.Net.Http.WinHttpResponseStream.CopyToAsyncCore(Stream destination, Byte[] buffer, CancellationToken cancellationToken)
/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncMethodBuilderCore.cs(42,0): at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at System.Net.Http.WinHttpResponseStream.CopyToAsyncCore(Stream destination, Byte[] buffer, CancellationToken cancellationToken)
/_/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpResponseStream.cs(106,0): at System.Net.Http.WinHttpResponseStream.CopyToAsync(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
/_/src/libraries/Common/src/System/Net/Http/NoWriteNoSeekStreamContent.cs(49,0): at System.Net.Http.NoWriteNoSeekStreamContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
/_/src/libraries/Common/src/System/Net/Http/NoWriteNoSeekStreamContent.cs(31,0): at System.Net.Http.NoWriteNoSeekStreamContent.SerializeToStreamAsync(Stream stream, TransportContext context)
/_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs(325,0): at System.Net.Http.HttpContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
/_/src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs(425,0): at System.Net.Http.HttpContent.LoadIntoBufferAsync(Int64 maxBufferSize, CancellationToken cancellationToken)
/_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(551,0): at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts, CancellationToken callerToken, Int64 timeoutTime)
/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs(290,0): at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs(172,0): at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs(329,0): at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs(307,0): at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs(784,0): at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(3236,0): at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(3208,0): at System.Threading.Tasks.Task.FinishContinuations()
/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Future.cs(398,0): at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskCompletionSource.cs(266,0): at System.Threading.Tasks.TaskCompletionSource`1.TrySetResult(TResult result)
/_/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpHandler.cs(898,0): at System.Net.Http.WinHttpHandler.StartRequestAsync(WinHttpRequestState state)
/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs(290,0): at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs(172,0): at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs(329,0): at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncTaskMethodBuilderT.cs(307,0): at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext()
/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(2364,0): at System.Threading.Tasks.Task.InnerInvoke()
/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(2352,0): at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs(289,0): at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(2314,0): at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(2246,0): at System.Threading.Tasks.Task.ExecuteEntryUnsafe(Thread threadPoolThread)
/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs(2237,0): at System.Threading.Tasks.Task.ExecuteFromThreadPool(Thread threadPoolThread)
/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs(659,0): at System.Threading.ThreadPoolWorkQueue.Dispatch()
/_/src/coreclr/src/System.Private.CoreLib/src/System/Threading/ThreadPool.CoreCLR.cs(29,0): at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
--- End of stack trace from AddCurrentStack ---
Seen in PR https://github.com/dotnet/runtime/pull/34077
@maryamariyan Yes, it seems to be the same issue:
WinHttp.WinHttpQueryDataAvailable
in System.Net.Http.WinHttpResponseStream.CopyToAsyncCore
ending with Error 12030 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'The connection with the server was terminated abnormally'
So far failing methods are (according to Kusto):
GetAsync_AddCookieHeader_CookieHeaderSent
GetAsync_AddMultipleCookieHeaders_CookiesSent
GetAsync_ReceiveInvalidSetCookieHeader_ValidCookiesAdded
GetAsync_ReceiveSetCookieHeader_CookieAdded
GetAsync_ReceiveSetCookieHeader_CookieUpdated
GetAsync_SetCookieContainer_CookieSent
GetAsyncWithRedirect_ReceiveSetCookie_CookieSent
GetAsyncWithRedirect_SetCookieContainer_CorrectCookiesSent
All above only for System.Net.Http.Functional.Tests.PlatformHandlerTest_Cookies_Http2
.
I'll disable all PlatformHandlerTest_Cookies_Http2
for WinHttpHandler
in this PR: https://github.com/dotnet/runtime/pull/34119
Looking at @ManickaP's traces with help from the WinHttp team we found out that the (loopback test) server closes the connection (sends FIN) before the client has chance to process the data -- in which case WinHttp discards all buffers, causing this test failure.
Hit in #33929
From https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-33929-merge-7da746c0d9ce477694/System.Net.Http.WinHttpHandler.Functional.Tests/console.d112d952.log?sv=2019-02-02&se=2020-04-01T14%3A48%3A02Z&sr=c&sp=rl&sig=KCetDHpe2GT7vLy78WQFBRcl8gD86%2BQ7pMCyvM6Lpvo%3D :