System.Net.*.Tests are unstable or slow/long-running on a S.P.CoreLib checked runtime

[safern]

We're disabling some System.Net.*.Tests assemblies when running on a checked runtime in: #261. However we should investigate what can we do to enable these tests there.

I.e System.Net.Sockets.Tests would be a really would candidate for runtime testing and stress testing.

cc: @BruceForstall @jkotas @stephentoub

[davidsh]

@wfurt

[ViktorHofer]

Do we know why especially the networking tests are so slow on a checked runtime?

[davidsh]

What is a "checked" runtime? I know what Debug vs. Release is. How does "checked" related to that?

[BruceForstall]

Checked is the same as Debug (e.g., asserts enabled), but with the C++ compiler optimization enabled (so native code runs much faster).

[wfurt]

It would be great to get more specific instructions how to know produce one and to know if test run is checked. I always assumed that checked would be slower than Debug but it seems like it may not. Lot of networking code (product & tests) depends on timing. That is different from let say data structure tests. There are some constants for test and AFAIK we already adjusted some for ARM runs. We could possibly do that for checked builds if needed. But first we need to be able to reproduce it local and take a look.

[karelz]

Any update on this? @BruceForstall @ViktorHofer @safern

[safern]

Here is a Checked run, where this tests had to be killed because they were long running:

https://helix.dot.net/api/2019-06-17/jobs/70eeb0e3-b877-4b51-a91b-2b5d8a69f8a8/workitems/System.Net.Mail.Functional.Tests/console

[wfurt]

console copy before link expires. I don't see any dump so this may be tricky. Also I noticed then when I'm running tests with debug CLR it is super slow even on my dev box.

C:\h\w\B22E09AD\w\A68A095C\e>"C:\h\w\B22E09AD\p\dotnet.exe" exec --runtimeconfig System.Net.Mail.Functional.Tests.runtimeconfig.json --depsfile System.Net.Mail.Functional.Tests.deps.json xunit.console.dll System.Net.Mail.Functional.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing -notrait category=nonnetcoreapptests -notrait category=nonwindowstests  
  Discovering: System.Net.Mail.Functional.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Mail.Functional.Tests (found 144 of 146 test cases)
  Starting:    System.Net.Mail.Functional.Tests (parallel test collections = on, max threads = 2)
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestCredentialsCopyInAsyncContext', Elapsed: 00:02:14
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestCredentialsCopyInAsyncContext', Elapsed: 00:04:15
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestCredentialsCopyInAsyncContext', Elapsed: 00:06:15
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestCredentialsCopyInAsyncContext', Elapsed: 00:08:15
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestCredentialsCopyInAsyncContext', Elapsed: 00:10:15
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestCredentialsCopyInAsyncContext', Elapsed: 00:12:15
   System.Net.Mail.Functional.Tests: [Long Running Test] 'System.Net.Mail.Tests.SmtpClientTest.TestCredentialsCopyInAsyncContext', Elapsed: 00:14:15

[safern]

Don't run them against a debug CLR, because that is not optimized, you should run them against a checked CLR.

src\coreclr\build.cmd -checked && libraries.cmd /p:CoreCLRConfiguration=checked

If you need it here is the testhost which was used for this job and contains a checked runtime: https://helixde107v0xdeko0k025g8.blob.core.windows.net/helix-job-515f0f7b-e695-4ee6-ac03-c600888962748843851101e463ebc/test-runtime-netcoreapp5.0-Windows_NT-Debug-x64.zip?sv=2018-03-28&sr=c&sig=Alo7OiJnlWnLBhPyRA8D2BDtKjmbjTiZg51IcVgk99o%3D&se=2020-01-25T23%3A52%3A54Z&sp=rl

And here's the test payload https://helixde107v0xdeko0k025g8.blob.core.windows.net/helix-job-515f0f7b-e695-4ee6-ac03-c600888962748843851101e463ebc/System.Net.Mail.Functional.Tests.zip?sv=2018-03-28&sr=c&sig=Alo7OiJnlWnLBhPyRA8D2BDtKjmbjTiZg51IcVgk99o%3D&se=2020-01-25T23%3A52%3A54Z&sp=rl

[dotnet-policy-service[bot]]

Due to lack of recent activity, this issue has been marked as a candidate for backlog cleanup. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will undo this process.

This process is part of our issue cleanup automation.