Open ghost opened 7 years ago
[EDIT] Test failure included in history in top post.
Updated test failure history in top post.
Seen in https://github.com/dotnet/corefx/pull/35267
Debian.8.Amd64.Open-x64-Release
Message :
System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known
Stack Trace :
at System.Net.Dns.InternalGetHostByName(String hostName) in /__w/1/s/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 68
at System.Net.Dns.GetHostByName(String hostName) in /__w/1/s/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 41
at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName() in /__w/1/s/src/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 108
Same test failed with similar exception:
Failed test:
System.Net.NameResolution.Tests.GetHostByNameTest/DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName
Message:
System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known
Stack Trace :
at System.Net.Dns.InternalGetHostByName(String hostName) in /_/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 68
at System.Net.Dns.ResolveCallback(Object context) in /_/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 218
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source) in /_/src/System.Private.CoreLib/shared/System/Runtime/ExceptionServices/ExceptionDispatchInfo.cs:line 69
at System.Net.Dns.HostResolutionEndHelper(IAsyncResult asyncResult) in /_/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 361
at System.Net.Dns.EndGetHostByName(IAsyncResult asyncResult) in /_/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 383
at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName() in /_/src/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 119
Build: -20190617.81(Master) Failing configurations:
Triage: We should know why DNS resolution does not work sometimes -- we know it is not cached on Linux/Mac, but why does it happen so often in CI? Is it network flakiness? Or busy machine?
Plan: Add production diagnostics into DNS, then make runs with those changes to catch it. It will be useful also for production diagnostic by customers.
no failure on Linux for 90 days. Some tests are still disabled on OSX and ARM64 (tracked by #27622).
@wfurt do we have any tests disabled against this specific bug? If not, let's close it.
We have four tests disabled at this moment for OSX and the same for ARM64 (tracked in #27622).
I suspect most of the failures are not product bugs and are caused by test environment.
AFAIK we changed our macOS infrastructure as well as ARM64 support matured.
How about moving the test to Outerloop? That would allow us to collect some stability data without impacting everyone's PRs. That also seems appropriate since tests do have a dependency on external service.
cc: @dotnet/ncl in case somebody has a better suggestion.
A bunch of NameResolution tests failed again (including the ones mentioned in this issue): https://dnceng.visualstudio.com/public/_build/results?buildId=525372&view=ms.vss-test-web.build-test-results-tab&runId=16621996&resultId=137716&paneView=debug
@wfurt @davish suggestions for mitigation?
@wfurt @davish suggestions for mitigation?
Doing name resolution in a CI environment will always be flaky most likely.
System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known
Perhaps we should consider an approach being done here in #32501 and only run these tests on a less regular basis. I.e.:
[Trait(XunitConstants.Category, XunitConstants.IgnoreForCI)] // DNS is flaky
FWIW, Our DNS APIs have been relatively stable from a product perspective and don't see a lot of risk running these tests less often.
We should work towards stabilizing core services (like DNS) in CI.
What about leveraging the RetryHelper
for DNS tests which are inherently flaky?
We should work towards stabilizing core services (like DNS) in CI.
What about leveraging the RetryHelper for DNS tests which are inherently flaky?
Do we understand whether the problem is intermittent (i.e. retrying tests will help) or due to a mis-configuration of some kind in CI (which retrying tests will not help).
Do we understand whether the problem is intermittent (i.e. retrying tests will help) or due to a mis-configuration of some kind in CI (which retrying tests will not help).
I assumed it's intermittent but how would we find that out?
I assumed it's intermittent but how would we find that out?
Let's add some RetryHelper logic to some of the tests. Then run it in CI. If we see a significant improvement in test reliability, then we will know that the problem is intermittent.
Tests were disabled on SLES with https://github.com/dotnet/runtime/pull/48759.
Failed again in runtime 20210624.123
Failed tests:
net6.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open
- System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntry_HostString_Ok(hostName: \"\")
Error message
System.Exception : Failed for empty hostname.
Dns.GetHostName() == a000AID
getHostEntryException==System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in //src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in //src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
at System.Net.Dns.GetHostEntry(String hostNameOrAddress, AddressFamily family) in //src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 91
at System.Net.Dns.GetHostEntry(String hostNameOrAddress) in //src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 60
at System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntry_HostString_Ok(String hostName) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 45
etcHostsException==
/etc/host ==
hosts This file describes a number of hostname-to-address
mappings for the TCP/IP subsystem. It is mostly
used at boot time, when no name servers are running.
On small systems, this file can be used instead of a
"named" name server.
Syntax:
IP-Address Full-Qualified-Hostname Short-Hostname
127.0.0.1 localhost
special IPv6 addresses
::1 localhost ipv6-localhost ipv6-loopback
fe00::0 ipv6-localnet
ff00::0 ipv6-mcastprefix
ff02::1 ipv6-allnodes
ff02::2 ipv6-allrouters
ff02::3 ipv6-allhosts
Added by SMT registration do not remove, retain comment as well
52.148.152.22 smt-azure.susecloud.net smt-azure
---- System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known
Stack trace
at System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntry_HostString_Ok(String hostName) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 64
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
at System.Net.Dns.GetHostEntry(String hostNameOrAddress, AddressFamily family) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 91
at System.Net.Dns.GetHostEntry(String hostNameOrAddress) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 60
at System.Net.NameResolution.Tests.GetHostEntryTest.<>c__DisplayClass1_0.<Dns_GetHostEntry_HostString_Ok>b__0() in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 33
at System.Net.NameResolution.Tests.GetHostEntryTest.TestGetHostEntryAsync(Func`1 getHostEntryFunc) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 88
at System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntry_HostString_Ok(String hostName) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 33
net6.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open
- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName
- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName
net6.0-Linux-Release-x64-Mono_release-SLES.12.Amd64.Open
- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName
- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName
Error message
System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known
Stack trace
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
at System.Net.Dns.GetHostByName(String hostName) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 271
at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName() in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 109
net6.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open
- System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntryAsync_HostString_Ok(hostName: \"\")
Error message
System.AggregateException : One or more errors occurred. (One or more errors occurred. (Name or service not known)) (One or more errors occurred. (Name or service not known))
---- System.AggregateException : One or more errors occurred. (Name or service not known)
-------- System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known
---- System.AggregateException : One or more errors occurred. (Name or service not known)
-------- System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known
Stack trace
at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 80
at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks, Int32 millisecondsTimeout) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 49
at System.Net.NameResolution.Tests.GetHostEntryTest.TestGetHostEntryAsync(Func`1 getHostEntryFunc) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 91
at System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntryAsync_HostString_Ok(String hostName) in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostEntryTest.cs:line 80
--- End of stack trace from previous location ---
----- Inner Stack Trace #1 (System.AggregateException) -----
----- Inner Stack Trace -----
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__33_1(Object s) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 591
at System.Net.Dns.<>c__DisplayClass39_0`1.<RunAsync>b__0(Task <p0>, Object <p1>) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 688
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 94
at System.Threading.Tasks.Task.<>c.<.cctor>b__284_0(Object obj) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2369
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 268
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 293
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2331
--- End of stack trace from previous location ---
at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 63
----- Inner Stack Trace #2 (System.AggregateException) -----
----- Inner Stack Trace -----
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__33_1(Object s) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 591
at System.Net.Dns.<>c__DisplayClass39_0`1.<RunAsync>b__0(Task <p0>, Object <p1>) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 688
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs:line 94
at System.Threading.Tasks.Task.<>c.<.cctor>b__284_0(Object obj) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2369
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 268
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs:line 293
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs:line 2331
Failed again in runtime 20210704.33
Failed tests:
net6.0-Linux-Release-x64-CoreCLR_release-SLES.12.Amd64.Open
- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName
- System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName
- System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntry_HostString_Ok(hostName: \"\")
- System.Net.NameResolution.Tests.GetHostEntryTest.Dns_GetHostEntryAsync_HostString_Ok(hostName: \"\")
One of Error messages:
System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Name or service not known
Stack trace
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 403
at System.Net.Dns.GetHostEntryCore(String hostName, AddressFamily addressFamily) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 384
at System.Net.Dns.GetHostByName(String hostName) in /_/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs:line 271
at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName() in /_/src/libraries/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 109
Type of failures
Affected tests:
System.Net.NameResolution.Tests.GetHostByNameTest
(both always fail together)DnsObsoleteGetHostByName_EmptyString_ReturnsHostName
DnsObsoleteBeginEndGetHostByName_EmptyString_ReturnsHostName
System.Net.NameResolution.Tests.GetHostEntryTest
(both always fail together)Dns_GetHostEntry_HostString_Ok
Dns_GetHostEntryAsync_HostString_Ok
Failure text:
at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6) in /Users/buildagent/agent/_work/34/s/corefx/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 46 at System.Net.Dns.GetHostByName(String hostName) in /Users/buildagent/agent/_work/34/s/corefx/src/System.Net.NameResolution/src/System/Net/DNS.cs:line 41 at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName() in /Users/buildagent/agent/_work/34/s/corefx/src/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 108
Assert.Contains() Failure Not found: DCI-Mac-Build-068.local In value: dci-mac-build-068.local at System.Net.NameResolution.Tests.GetHostByNameTest.DnsObsoleteGetHostByName_EmptyString_ReturnsHostName() in /Users/buildagent/agent/_work/30/s/corefx/src/System.Net.NameResolution/tests/FunctionalTests/GetHostByNameTest.cs:line 108