Test failure: Server_SetHttp503VebosittHittingThrottle_Success

[ryanbrandenburg]

This test fails occasionally with the following error:

System.Net.Http.HttpRequestException : Response status code does not indicate success: 503 (Service Unavailable).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at System.Net.Http.HttpClient.GetStringAsyncCore(Task`1 getTask)
   at Microsoft.AspNetCore.Server.HttpSys.ServerTests.SendRequestAsync(String uri) in /_/test/Microsoft.AspNetCore.Server.HttpSys.FunctionalTests/ServerTests.cs:line 638
   at Microsoft.AspNetCore.Server.HttpSys.ServerTests.Server_SetHttp503VebosittHittingThrottle_Success() in /_/test/Microsoft.AspNetCore.Server.HttpSys.FunctionalTests/ServerTests.cs:line 334
--- End of stack trace from previous location where exception was thrown ---

Other tests within that build may have failed with a similar message, but they are not listed here. Check the link above for more info.

This test failed on master.

CC @Eilon (because the bot doesn't know who else to pick)

This issue was made automatically. If there is a problem contact ryanbrandenburg.

[ryanbrandenburg]

Please use this workflow to address this flaky test issue, including checking applicable checkboxes and filling in the applicable "TODO" entries:

• Is this actually a flaky test?

  • No, this is a regular test failure, fix the test/product (TODO: Link to commit/PR)
  • Yes, proceed below...

• Is this test failure caused by product code flakiness? (Either this product, or another product this test depends on.)

  • [ ] File a bug against the product (TODO: Link to other bug)
  • Is it possible to change the test to avoid the flakiness?
  • Yes? Go to the "Change the test!" section.
  • No?
    • [ ] Disable the test (TODO: Link to PR/commit)
    • [ ] Wait for other bug to be resolved
    • [ ] Wait for us to get build that has the fix
    • [ ] Re-enable our test (TODO: Link to PR/commit)
    • [ ] Close this bug

• Is it that the test itself is flaky? This includes external transient problems (e.g. remote server problems, file system race condition, etc.)

  • Is there is a way to change our test to avoid this flakiness?
  • Yes? Change the test!
    • [ ] Change the test to avoid the flakiness, for example by using a different test strategy, or by adding retries w/ timeouts (TODO: Link to PR/commit)
    • [ ] Run the test 100 times locally as a sanity check.
    • [ ] Close this bug
  • No?
    • Is there any logging or extra information that we could add to make this more diagnosable when it happens again?
    • Yes?
      • [ ] Add the logging (TODO: Link to PR/commit)
    • No?
      • [ ] Delete the test because flaky tests are not useful (TODO: Link to PR/commit)

This comment was made automatically. If there is a problem contact ryanbrandenburg.

[Tratcher]

Note the exception comes from here: https://github.com/aspnet/HttpSysServer/blob/eba2d7e380fef9f75203bb9a1f4d30827451d512/test/Microsoft.AspNetCore.Server.HttpSys.FunctionalTests/ServerTests.cs#L334

It's trying to verify the server recovers after some connections are closed. No indication of why the server did not recover in time.

4 failures in recorded history ranging between April and August, all on different OS's/agents.

[ryanbrandenburg]

@muratg was this resolved? Please assign someone to find out.

[aspnet-hello]

Server_SetHttp503VebosittHittingThrottle_Success failed with about the same error on master.

This comment was made automatically. If there is a problem contact ryanbrandenburg.

[aspnet-hello]

There were 1 failures with about the same error on master:

• Server_SetHttp503VebosittHittingThrottle_Success

This comment was made automatically. If there is a problem contact ryanbrandenburg.

[Eilon]

@Tratcher looks like this one is still being relatively flaky, so please try to come up with a solution for this.

[aspnet-hello]

This comment was made automatically. If there is a problem contact ryanbrandenburg.

There were 1 failures with about the same error on master at 10:47:18 AM:

• Server_SetHttp503VebosittHittingThrottle_Success

[Tratcher]

No repro, but I did do some code cleanup that should help.