search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.25k stars 4.73k forks source link

HttpListener tests timing out on Mono in CI #2391

Open stephentoub opened 4 years ago

stephentoub commented 4 years ago

https://helix.dot.net/api/2019-06-17/jobs/50e7502e-a749-4812-a7f0-e5edf3eeba61/workitems/System.Net.HttpListener.Tests/console

  Discovering: System.Net.HttpListener.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.HttpListener.Tests (found 198 of 298 test cases)
  Starting:    System.Net.HttpListener.Tests (parallel test collections = on, max threads = 2)
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.DrainEntityBody_Roundtrips [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.UnsupportedProperties_Throw [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.IdleConnection_Roundtrips [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.Properties_DefaultValues [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.Dispose_CallAfterDisposed_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.ReceiveAsync_Disposed_ThrowsObjectDisposedException [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_AfterDisposed_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseOutputAsync_HandshakeStartedFromClient_Success [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_AfterAborted_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_HandshakeStartedFromClient_Success [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.SendAsync_Disposed_ThrowsObjectDisposedException [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpRequestStreamTests.Read_FromClosedConnectionAsynchronously_ThrowsHttpListenerException', Elapsed: 00:02:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpRequestStreamTests.Read_FromClosedConnectionAsynchronously_ThrowsHttpListenerException', Elapsed: 00:04:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpRequestStreamTests.Read_FromClosedConnectionAsynchronously_ThrowsHttpListenerException', Elapsed: 00:06:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpRequestStreamTests.Read_FromClosedConnectionAsynchronously_ThrowsHttpListenerException', Elapsed: 00:08:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpRequestStreamTests.Read_FromClosedConnectionAsynchronously_ThrowsHttpListenerException', Elapsed: 00:10:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpRequestStreamTests.Read_FromClosedConnectionAsynchronously_ThrowsHttpListenerException', Elapsed: 00:12:04

cc: @akoeplinger

akoeplinger commented 4 years ago

I've tried reproducing these System.Net.HttpListener.Tests timeouts but wasn't able to, even using the Docker containers on my Mac.

@directhex can you please try to repro and/or disable these tests?

stephentoub commented 4 years ago

Another timeout, different PR, different test: https://helix.dot.net/api/2019-06-17/jobs/01ad5712-0a89-4684-9e9d-b80612973f2e/workitems/System.Net.HttpListener.Tests/console

  Discovering: System.Net.HttpListener.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.HttpListener.Tests (found 198 of 298 test cases)
  Starting:    System.Net.HttpListener.Tests (parallel test collections = on, max threads = 2)
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.DrainEntityBody_Roundtrips [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.UnsupportedProperties_Throw [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.IdleConnection_Roundtrips [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.Properties_DefaultValues [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.Dispose_CallAfterDisposed_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.ReceiveAsync_Disposed_ThrowsObjectDisposedException [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_AfterDisposed_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseOutputAsync_HandshakeStartedFromClient_Success [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_AfterAborted_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_HandshakeStartedFromClient_Success [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.SendAsync_Disposed_ThrowsObjectDisposedException [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.SendAsync_NoInnerBuffer_ThrowsArgumentNullException', Elapsed: 00:02:05
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.SendAsync_NoInnerBuffer_ThrowsArgumentNullException', Elapsed: 00:04:05
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.SendAsync_NoInnerBuffer_ThrowsArgumentNullException', Elapsed: 00:06:05
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.SendAsync_NoInnerBuffer_ThrowsArgumentNullException', Elapsed: 00:08:05
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.SendAsync_NoInnerBuffer_ThrowsArgumentNullException', Elapsed: 00:10:05
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.SendAsync_NoInnerBuffer_ThrowsArgumentNullException', Elapsed: 00:12:05
directhex commented 4 years ago

@akoeplinger what's the quickest way to try and repro?

stephentoub commented 4 years ago

Another, again different PR, different test: https://helix.dot.net/api/2019-06-17/jobs/5a44346f-0d12-4dc5-9847-8e3ddda83ed6/workitems/System.Net.HttpListener.Tests/console

  Discovering: System.Net.HttpListener.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.HttpListener.Tests (found 198 of 298 test cases)
  Starting:    System.Net.HttpListener.Tests (parallel test collections = on, max threads = 2)
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.DrainEntityBody_Roundtrips [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.UnsupportedProperties_Throw [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.IdleConnection_Roundtrips [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.Properties_DefaultValues [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.Dispose_CallAfterDisposed_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.ReceiveAsync_Disposed_ThrowsObjectDisposedException [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_AfterDisposed_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseOutputAsync_HandshakeStartedFromClient_Success [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_AfterAborted_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_HandshakeStartedFromClient_Success [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.SendAsync_Disposed_ThrowsObjectDisposedException [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpResponseStreamTests.Write_TooLittleSynchronouslyAndClose_ThrowsInvalidOperationException [SKIP]
      Condition(s) not met: "IsWindowsImplementation"
    System.Net.Tests.HttpResponseStreamTests.Write_TooLittleAsynchronouslyAndClose_ThrowsInvalidOperationException [SKIP]
      Condition(s) not met: "IsWindowsImplementation"
    System.Net.Tests.HttpResponseStreamTests.Write_TooMuch_ThrowsProtocolViolationException [SKIP]
      Condition(s) not met: "IsWindowsImplementation"
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpResponseStreamTests.EndWrite_CalledTwice_ThrowsInvalidOperationException', Elapsed: 00:02:03
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpResponseStreamTests.EndWrite_CalledTwice_ThrowsInvalidOperationException', Elapsed: 00:04:03
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpResponseStreamTests.EndWrite_CalledTwice_ThrowsInvalidOperationException', Elapsed: 00:06:03
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpResponseStreamTests.EndWrite_CalledTwice_ThrowsInvalidOperationException', Elapsed: 00:08:03
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpResponseStreamTests.EndWrite_CalledTwice_ThrowsInvalidOperationException', Elapsed: 00:10:03
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpResponseStreamTests.EndWrite_CalledTwice_ThrowsInvalidOperationException', Elapsed: 00:12:03
akoeplinger commented 4 years ago

@directhex The way I did was to look at e.g. https://helix.dot.net/api/2019-06-17/jobs/01ad5712-0a89-4684-9e9d-b80612973f2e/workitems/System.Net.HttpListener.Tests and download the run_client.py log which contains the various payloads, then I launched a docker container, downloaded and unzipped all the payloads and used the RunTests.sh from there.

directhex commented 4 years ago

Starting: System.Net.HttpListener.Tests (parallel test collections = on, max threads = 8)

directhex commented 4 years ago

Which Docker image do we use to run this on Helix? I want my runs to match where it's failing as closely as possible

stephentoub commented 4 years ago

Another failure here, though not manifesting in the same way (so not sure if it's actually the same or not): https://helix.dot.net/api/2019-06-17/jobs/b9a45ad2-d9fb-455a-a6e6-e66dc29cf73b/workitems/System.Net.HttpListener.Tests/console

    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_InvalidCloseStatus_ThrowsArgumentException(status: EndpointUnavailable, statusDescription: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"..., paramName: "statusDescription") [FAIL]
      System.Net.WebSockets.WebSocketException : The server returned status code '400' when status code '101' was expected.
      Stack Trace:
           at System.Net.WebSockets.WebSocketHandle.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)
           at System.Net.WebSockets.ClientWebSocket.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken)
           at System.Net.Tests.HttpListenerWebSocketTests.GetWebSocketContext(String[] subProtocols)
           at System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_InvalidCloseStatus_ThrowsArgumentException(WebSocketCloseStatus status, String statusDescription, String paramName)
        --- End of stack trace from previous location ---
directhex commented 4 years ago

I've just run these tests 40 times without reproducing the problem :man_shrugging:

danmoseley commented 4 years ago

What configuration did you use @directhex ?

Eg., I've seen it on netcoreapp5.0-Linux-Debug-x64-Mono_release-Centos.7.Amd64.Open

@safern can tell you the docker image.

danmoseley commented 4 years ago

https://dev.azure.com/dnceng/public/_build/results?buildId=502018&view=ms.vss-test-web.build-test-results-tab&runId=15945120&resultId=168480&paneView=debug is another instance, it's on netcoreapp5.0-Linux-Debug-x64-Mono_release-(Fedora.29.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-29-helix-a12566d-20191210224553

safern commented 4 years ago

I just looked at all the links that @stephentoub pasted, none of them ran in a docker container, I found that there are 3 different flavors that hit this issue:

Centos7 Debian9 SLES15

Maybe that's why you can't repro?

This is the test dll and its dependency for one of the jobs above: https://helixde107v0xdeko0k025g8.blob.core.windows.net/helix-job-4c5b489e-62be-4fc0-8eaa-319efce3de0d30db8ad0257421b8f/System.Net.HttpListener.Tests.zip?sv=2019-02-02&se=2020-02-09T15%3A10%3A50Z&sr=c&sp=rl&sig=KC2ck%2BrOaDkIC7SJoLu%2BtY4tqbwpUsnrqJju9t34%2BOQ%3D

And this is the correlation payload (testhost) https://helixde107v0xdeko0k025g8.blob.core.windows.net/helix-job-4c5b489e-62be-4fc0-8eaa-319efce3de0d30db8ad0257421b8f/test-runtime-netcoreapp5.0-Linux-Debug-x64.zip?sv=2019-02-02&se=2020-02-09T15%3A10%3A50Z&sr=c&sp=rl&sig=KC2ck%2BrOaDkIC7SJoLu%2BtY4tqbwpUsnrqJju9t34%2BOQ%3D

That one ran on: sles.15.amd64.open

stephentoub commented 4 years ago

https://helix.dot.net/api/2019-06-17/jobs/d8881f6f-0bc3-4c80-8400-d1e445618ad8/workitems/System.Net.HttpListener.Tests/console

  Discovering: System.Net.HttpListener.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.HttpListener.Tests (found 198 of 298 test cases)
  Starting:    System.Net.HttpListener.Tests (parallel test collections = on, max threads = 2)
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.DrainEntityBody_Roundtrips [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.UnsupportedProperties_Throw [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.IdleConnection_Roundtrips [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerTimeoutManagerUnixTests.Properties_DefaultValues [SKIP]
      Condition(s) not met: "IsManagedImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.Dispose_CallAfterDisposed_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.ReceiveAsync_Disposed_ThrowsObjectDisposedException [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_AfterDisposed_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseOutputAsync_HandshakeStartedFromClient_Success [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_AfterAborted_Nop [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_HandshakeStartedFromClient_Success [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
    System.Net.Tests.HttpListenerWebSocketTests.SendAsync_Disposed_ThrowsObjectDisposedException [SKIP]
      Condition(s) not met: "IsNotWindows7AndIsWindowsImplementation"
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_InvalidCloseStatus_ThrowsArgumentException', Elapsed: 00:02:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_InvalidCloseStatus_ThrowsArgumentException', Elapsed: 00:04:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_InvalidCloseStatus_ThrowsArgumentException', Elapsed: 00:06:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_InvalidCloseStatus_ThrowsArgumentException', Elapsed: 00:08:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_InvalidCloseStatus_ThrowsArgumentException', Elapsed: 00:10:04
   System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerWebSocketTests.CloseAsync_InvalidCloseStatus_ThrowsArgumentException', Elapsed: 00:12:04