dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.88k stars 4.63k forks source link

Tests failures: System.Net.NameResolution.Tests / DnsObsolete* & Dns_GetHostEntry* #1488

Open ghost opened 6 years ago

ghost commented 6 years ago

Type of failures

Affected tests:

Failure text:

at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6) in /Users/buildagent/agent/_work/34/s/corefx/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 46 at System.Net.Dns.GetHostByName(String hostName) in /Users/buildagent/agent/_work/34/s/corefx/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 41 at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName() in /Users/buildagent/agent/_work/34/s/corefx/src/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 108


Note:
 * `Dns_GetHostEntryAsync_HostString_Ok` fails almost always with same error twice, wrapped in `System.AggregateException`.

### Older failure on OSX (prior to 2017/9/15)

Assert.Contains() Failure Not found: DCI-Mac-Build-068.local In value: dci-mac-build-068.local at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName() in /Users/buildagent/agent/_work/30/s/corefx/src/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 108



## History of failures

Each test couple always fails or passes - maybe bad machine setup?

Day | Build | OS | Test
-- | -- | -- | --
9/6 | 20170906.01 | OSX10.12 | DnsObsolete*
9/10 | 20170910.01 | Ubuntu14.04 | DnsObsolete* & Dns_GetHostEntry*
9/13 | 20170913.02 | OSX10.12 | DnsObsolete*
9/15 | 20170915.01 | OSX10.12 | DnsObsolete*
9/20 | 20170920.03 | Debian90 | DnsObsolete* & Dns_GetHostEntry*
9/30 | 20170930.01 | OSX10.12 | DnsObsolete* & Dns_GetHostEntry*
10/2 | 20171002.01 | Debian87 | DnsObsolete* & Dns_GetHostEntry*
10/21 | 20171021.01 | Debian87 | DnsObsolete* & Dns_GetHostEntry*
10/26 | 20171026.01 | Suse42.2 | DnsObsolete* & Dns_GetHostEntry*
11/13 | 20171113.03 | OSX10.12 | DnsObsolete* & Dns_GetHostEntry*
11/16 | 20171116.51 | SLES12 | DnsObsolete* & Dns_GetHostEntry*
11/26 | 20171126.02 | Suse42.2 | DnsObsolete* & Dns_GetHostEntry*

Similar failures to dotnet/corefx#20245, but at different times (probably runs on different worker machine).
<!-- runfo report start -->
Runfo Tracking Issue: [dnsobsoletegethostbyname_emptystring_returnshostname](https://runfo.azurewebsites.net/tracking/issue/48)
|Build|Definition|Kind|Run Name|
|---|---|---|---|

Build Result Summary
|Day Hit Count|Week Hit Count|Month Hit Count|
|---|---|---|
|0|0|0|

<!-- runfo report end -->
Sunny-pu commented 6 years ago

[EDIT] Test failure included in history in top post.

karelz commented 6 years ago

Updated test failure history in top post.

Sunny-pu commented 6 years ago

failed again: https://mc.dot.net/#/product/netcore/master/source/official~2Fcorefx~2Fmaster~2F/type/test~2Ffunctional~2Fcli~2F/build/20180419.01/workItem/System.Net.NameResolution.Functional.Tests

benaadams commented 5 years ago

Seen in https://github.com/dotnet/corefx/pull/35267

Debian.8.Amd64.Open-x64-Release

Message :
System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known
Stack Trace :
   at System.Net.Dns.InternalGetHostByName(String hostName) in /__w/1/s/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 68
   at System.Net.Dns.GetHostByName(String hostName) in /__w/1/s/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 41
   at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName() in /__w/1/s/src/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 108
AriNuer commented 5 years ago

Same test failed with similar exception:

Failed test: System.Net.NameResolution.Tests.GetHostByNameTest/DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName

Message:

System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known

Stack Trace :

   at System.Net.Dns.InternalGetHostByName(String hostName) in /_/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 68
   at System.Net.Dns.ResolveCallback(Object context) in /_/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 218
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source) in /_/src/System.Private.CoreLib/shared/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 69
   at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult) in /_/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 361
   at System.Net.Dns.EndGetHostByName(IAsyncResult asyncResult) in /_/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 383
   at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName() in /_/src/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 119

Build: -20190617.81(Master) Failing configurations:

Details: https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fheads~2Fmaster/test~2Ffunctional~2Fcli~2Finnerloop~2F/20190617.81/workItem/System.Net.NameResolution.Functional.Tests/analysis/xunit/System.Net.NameResolution.Tests.GetHostByNameTest~2FDnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName

karelz commented 4 years ago

Triage: We should know why DNS resolution does not work sometimes -- we know it is not cached on Linux/Mac, but why does it happen so often in CI? Is it network flakiness? Or busy machine?

Plan: Add production diagnostics into DNS, then make runs with those changes to catch it. It will be useful also for production diagnostic by customers.

wfurt commented 4 years ago

no failure on Linux for 90 days. Some tests are still disabled on OSX and ARM64 (tracked by #27622).

karelz commented 4 years ago

@wfurt do we have any tests disabled against this specific bug? If not, let's close it.

wfurt commented 4 years ago

We have four tests disabled at this moment for OSX and the same for ARM64 (tracked in #27622).

I suspect most of the failures are not product bugs and are caused by test environment.
AFAIK we changed our macOS infrastructure as well as ARM64 support matured. How about moving the test to Outerloop? That would allow us to collect some stability data without impacting everyone's PRs. That also seems appropriate since tests do have a dependency on external service.

cc: @dotnet/ncl in case somebody has a better suggestion.

ViktorHofer commented 4 years ago

A bunch of NameResolution tests failed again (including the ones mentioned in this issue): https://dnceng.visualstudio.com/public/_build/results?buildId=525372&view=ms.vss-test-web.build-test-results-tab&runId=16621996&resultId=137716&paneView=debug

@wfurt @davish suggestions for mitigation?

davidsh commented 4 years ago

@wfurt @davish suggestions for mitigation?

Doing name resolution in a CI environment will always be flaky most likely.

System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known

Perhaps we should consider an approach being done here in #32501 and only run these tests on a less regular basis. I.e.:

[Trait(XunitConstants.Category, XunitConstants.IgnoreForCI)] // DNS is flaky

FWIW, Our DNS APIs have been relatively stable from a product perspective and don't see a lot of risk running these tests less often.

wfurt commented 4 years ago

We should work towards stabilizing core services (like DNS) in CI.

ViktorHofer commented 4 years ago

What about leveraging the RetryHelper for DNS tests which are inherently flaky?

davidsh commented 4 years ago

We should work towards stabilizing core services (like DNS) in CI.

What about leveraging the RetryHelper for DNS tests which are inherently flaky?

Do we understand whether the problem is intermittent (i.e. retrying tests will help) or due to a mis-configuration of some kind in CI (which retrying tests will not help).

ViktorHofer commented 4 years ago

Do we understand whether the problem is intermittent (i.e. retrying tests will help) or due to a mis-configuration of some kind in CI (which retrying tests will not help).

I assumed it's intermittent but how would we find that out?

davidsh commented 4 years ago

I assumed it's intermittent but how would we find that out?

Let's add some RetryHelper logic to some of the tests. Then run it in CI. If we see a significant improvement in test reliability, then we will know that the problem is intermittent.

Anipik commented 3 years ago

failed again here https://dev.azure.com/dnceng/public/_build/results?buildId=1011268&view=logs&j=b9b7de4d-2126-5581-6453-1576f9aaac5f&t=db048d48-40e8-5399-9b14-d587763f33ff

eerhardt commented 3 years ago

Here too: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-48667-merge-af809879612c4371b8/System.Net.NameResolution.Functional.Tests/console.826d39cc.log?sv=2019-07-07&se=2021-03-17T22%3A53%3A24Z&sr=c&sp=rl&sig=qfUTGHjOCkAigQRcROLrLRTtq5MphSvppEvcO8QVJUU%3D

ViktorHofer commented 3 years ago

Tests were disabled on SLES with https://github.com/dotnet/runtime/pull/48759.

VincentBu commented 3 years ago

Failed again in runtime 20210624.123

Failed tests:

net6.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open

- System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntry_HostString_Ok(hostName: \"\")

Error message
System.Exception : Failed for empty hostname.
Dns.GetHostName() == a000AID
getHostEntryException==System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in //src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in //src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
at System.Net.Dns.GetHostEntry(String hostNameOrAddress, AddressFamily family) in //src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 91
at System.Net.Dns.GetHostEntry(String hostNameOrAddress) in //src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 60
at System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntry_HostString_Ok(String hostName) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 45
etcHostsException==
/etc/host ==

hosts         This file describes a number of hostname-to-address
mappings for the TCP/IP subsystem.  It is mostly
used at boot time, when no name servers are running.
On small systems, this file can be used instead of a
"named" name server.
Syntax:
IP-Address  Full-Qualified-Hostname  Short-Hostname
127.0.0.1   localhost

special IPv6 addresses
::1             localhost ipv6-localhost ipv6-loopback

fe00::0         ipv6-localnet

ff00::0         ipv6-mcastprefix
ff02::1         ipv6-allnodes
ff02::2         ipv6-allrouters
ff02::3         ipv6-allhosts

Added by SMT registration do not remove, retain comment as well
52.148.152.22   smt-azure.susecloud.net smt-azure

---- System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known

Stack trace
   at System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntry_HostString_Ok(String hostName) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 64
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
   at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
   at System.Net.Dns.GetHostEntry(String hostNameOrAddress, AddressFamily family) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 91
   at System.Net.Dns.GetHostEntry(String hostNameOrAddress) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 60
   at System.Net.NameResolution.Tests.GetHostEntryTest.<>c__DisplayClass1_0.<Dns_GetHostEntry_HostString_Ok>b__0() in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 33
   at System.Net.NameResolution.Tests.GetHostEntryTest.TestGetHostEntryAsync(Func`1 getHostEntryFunc) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 88
   at System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntry_HostString_Ok(String hostName) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 33
net6.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open

- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName
- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName

net6.0-Linux-Release-x64-Mono_release-SLES.12.Amd64.Open

- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName
- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName

Error message
System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known

Stack trace
   at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
   at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
   at System.Net.Dns.GetHostByName(String hostName) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 271
   at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName() in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 109
net6.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open

- System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntryAsync_HostString_Ok(hostName: \"\")

Error message
System.AggregateException : One or more errors occurred. (One or more errors occurred. (Name or service not known)) (One or more errors occurred. (Name or service not known))
---- System.AggregateException : One or more errors occurred. (Name or service not known)
-------- System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known
---- System.AggregateException : One or more errors occurred. (Name or service not known)
-------- System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known

Stack trace
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 80
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 49
   at System.Net.NameResolution.Tests.GetHostEntryTest.TestGetHostEntryAsync(Func`1 getHostEntryFunc) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 91
   at System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntryAsync_HostString_Ok(String hostName) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 80
--- End of stack trace from previous location ---
----- Inner Stack Trace #1 (System.AggregateException) -----

----- Inner Stack Trace -----
   at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
   at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
   at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__33_1(Object s) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 591
   at System.Net.Dns.<>c__DisplayClass39_0`1.<RunAsync>b__0(Task <p0>, Object <p1>) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 688
   at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 94
   at System.Threading.Tasks.Task.<>c.<.cctor>b__284_0(Object obj) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2369
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 268
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 293
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2331
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 63
----- Inner Stack Trace #2 (System.AggregateException) -----

----- Inner Stack Trace -----
   at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
   at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
   at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__33_1(Object s) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 591
   at System.Net.Dns.<>c__DisplayClass39_0`1.<RunAsync>b__0(Task <p0>, Object <p1>) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 688
   at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 94
   at System.Threading.Tasks.Task.<>c.<.cctor>b__284_0(Object obj) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2369
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 268
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 293
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2331
VincentBu commented 3 years ago

Failed again in runtime 20210704.33

Failed tests:

net6.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open

- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName
- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName
- System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntry_HostString_Ok(hostName: \"\")
- System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntryAsync_HostString_Ok(hostName: \"\")

One of Error messages:

System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known

Stack trace
   at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
   at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
   at System.Net.Dns.GetHostByName(String hostName) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 271
   at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName() in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 109