Flaky test: Caching_SetTtlAndStatusCode_Cached

ryanbrandenburg commented 6 years ago

This test fails occasionally with the following error:

System.Exception : Failed to get second response for 412
---- System.Threading.Tasks.TaskCanceledException : The operation was canceled.
-------- System.IO.IOException : Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
------------ System.Net.Sockets.SocketException : The I/O operation has been aborted because of either a thread exit or an application request.
   at Microsoft.AspNetCore.Server.HttpSys.Listener.ResponseCachingTests.Caching_SetTtlAndStatusCode_Cached() in /_/test/Microsoft.AspNetCore.Server.HttpSys.FunctionalTests/Listener/ResponseCachingTests.cs:line 584
--- End of stack trace from previous location where exception was thrown ---
----- Inner Stack Trace -----
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.AspNetCore.Server.HttpSys.Listener.ResponseCachingTests.SendRequestAsync(String uri, String method, String extraHeader, String extraHeaderValue, HttpCompletionOption httpCompletionOption) in /_/test/Microsoft.AspNetCore.Server.HttpSys.FunctionalTests/Listener/ResponseCachingTests.cs:line 1163
   at Microsoft.AspNetCore.Server.HttpSys.Listener.ResponseCachingTests.Caching_SetTtlAndStatusCode_Cached() in /_/test/Microsoft.AspNetCore.Server.HttpSys.FunctionalTests/Listener/ResponseCachingTests.cs:line 580
----- Inner Stack Trace -----
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
----- Inner Stack Trace -----

Other tests within that build may have failed with a similar message, but they are not listed here. Check the link above for more info.

CC @Eilon,@muratg,@mkArtakMSFT

This issue was made automatically. If there is a problem contact @ryanbrandenburg.

ryanbrandenburg commented 6 years ago

Caching_SetTtlAndStatusCode_Cached failed with about the same error. Please use this workflow to address this flaky test issue, including checking applicable checkboxes and filling in the applicable "TODO" entries:

Is this actually a flaky test?
- No, this is a regular test failure, fix the test/product (TODO: Link to commit/PR)
- Yes, proceed below...
Is this test failure caused by product code flakiness? (Either this product, or another product this test depends on.)
- [ ] File a bug against the product (TODO: Link to other bug)
- Is it possible to change the test to avoid the flakiness?
- Yes? Go to the "Change the test!" section.
- No?
  - [ ] Disable the test (TODO: Link to PR/commit)
  - [ ] Wait for other bug to be resolved
  - [ ] Wait for us to get build that has the fix
  - [ ] Re-enable our test (TODO: Link to PR/commit)
  - [ ] Close this bug
Is it that the test itself is flaky? This includes external transient problems (e.g. remote server problems, file system race condition, etc.)
- Is there is a way to change our test to avoid this flakiness?
- Yes? Change the test!
  - [ ] Change the test to avoid the flakiness, for example by using a different test strategy, or by adding retries w/ timeouts (TODO: Link to PR/commit)
  - [ ] Run the test 100 times locally as a sanity check.
  - [ ] Close this bug
- No?
  - [ ] Delete the test because flaky tests are not useful (TODO: Link to PR/commit)

This comment was made automatically. If there is a problem contact @ryanbrandenburg.

Eilon commented 6 years ago

@JunTaoLuo can you take a first look at this flaky test issue?

ryanbrandenburg commented 6 years ago

Caching_SetTtlAndStatusCode_Cached failed with about the same error on master.

This comment was made automatically. If there is a problem contact ryanbrandenburg.

JunTaoLuo commented 6 years ago

Looks like this only affects Windows 7 and Windows Server 2008; we have 100% pass rate on all Windows NT versions 6.2 and higher. It looks like it's caused by flakiness in HttpClient.

Eilon commented 6 years ago

Hmm HttpClient flakiness, that's not good...

muratg commented 6 years ago

Looks related to https://github.com/dotnet/corefx/issues/30691. If so, this it's an issue with some Kestrel tests as well.

cc @halter73

halter73 commented 6 years ago

Based on the exception details, this very well could be the same corefx issue.

aspnet-hello commented 5 years ago

This comment was made automatically. If there is a problem contact ryanbrandenburg.

There were 1 failures with about the same error on master at 10:43:47 PM:

Caching_SetTtlAndStatusCode_Cached

halter73 commented 5 years ago

Specifically this is the dotnet/corefx#30691 issue. @Tratcher This is one reason that just disabling ConnectionNotClosedWhenClientSatisfiesMinimumDataRateGivenLargeResponseHeaders is unappealing. As you can see in aspnet/KestrelHttpServer#2777, this causes flakiness in nearly all of our functional tests that send any reasonably large amount of data. Where would we draw the line on which functional tests to disable?

halter73 commented 5 years ago

We could try using aspnet/Logging#886 with a very specific predicate to retry any test that we've ever observed failing this way.

aspnet-hello commented 5 years ago

This comment was made automatically. If there is a problem contact ryanbrandenburg.

There were 1 failures with about the same error on master at 10:37:37 AM:

Caching_UnsupportedRequestMethods_BypassCacheAndLeaveItIntact

There were 1 failures with about the same error on master at 1:17:34 PM:

Caching_SetTtlAndStatusCode_Cached

aspnet-hello commented 5 years ago

This comment was made automatically. If there is a problem contact ryanbrandenburg.

There were 1 failures with about the same error on master at 1:52:06 AM:

Caching_SetTtlAndStatusCode_Cached

JunTaoLuo commented 5 years ago

Looks like the last failure was after your changes in https://github.com/aspnet/HttpSysServer/pull/483. The tests are still failing on Win7: http://aspnetci/viewLog.html?buildId=591404&tab=buildResultsDiv&buildTypeId=XPlat_Windows_Win7_Universe. Can you take a look @Tratcher?

Tratcher commented 5 years ago

These tests have been re-written.

aspnet / HttpSysServer

Flaky test: Caching_SetTtlAndStatusCode_Cached #468