katiehockman
commented
5 years ago /cc @mikioh @brad-burch @ianlancetaylor
Open katiehockman opened 5 years ago
katiehockman
commented
5 years ago /cc @mikioh @brad-burch @ianlancetaylor
ianlancetaylor
commented
5 years ago 2018-06-13T18:06:14-537fb06/darwin-amd64-10_10 2018-06-13T18:06:14-537fb06/darwin-amd64-10_11 2018-06-26T16:40:09-8997ec1/darwin-amd64-10_10 2018-06-26T16:40:09-8997ec1/darwin-amd64-10_11 2018-06-26T16:40:09-8997ec1/darwin-amd64-race 2018-07-03T16:14:43-2ee6bfb/darwin-amd64-10_10 2018-07-03T16:14:43-2ee6bfb/darwin-amd64-10_12 2018-07-03T16:27:51-aad71d3/darwin-amd64-10_12 2018-07-03T17:37:40-5d4f047/darwin-386-10_11 2018-07-03T17:37:40-5d4f047/darwin-amd64-10_10 2018-07-03T17:37:40-5d4f047/darwin-amd64-10_11 2018-07-09T22:19:21-94076fe/darwin-amd64-10_10 2018-07-11T01:08:56-5e70b13/darwin-amd64-race 2018-07-27T16:29:52-b39fb9e/darwin-amd64-10_10 2018-08-17T15:34:50-876c6d1/darwin-386-10_11 2018-08-17T15:34:50-876c6d1/darwin-amd64-10_10 2018-08-17T15:34:50-876c6d1/darwin-amd64-10_11 2018-08-21T04:39:22-80fe2e6/darwin-amd64-race 2018-08-21T04:50:21-2a5df06/darwin-386-10_11 2018-08-21T04:55:04-187a41d/darwin-amd64-10_10 2018-08-21T04:55:04-187a41d/darwin-amd64-10_11 2018-08-21T14:08:12-e8daca4/darwin-amd64-10_10 2018-08-22T22:11:24-ccb70bd/darwin-386-10_11 2018-08-24T23:02:01-aacc891/darwin-amd64-10_10 2018-08-24T23:02:31-523dc5a/darwin-amd64-10_10 2018-08-30T19:36:15-a2a8396/darwin-amd64-10_10 2018-08-30T19:36:15-a2a8396/darwin-amd64-race 2018-10-03T15:01:47-c96e3bc/darwin-amd64-10_10 2018-10-03T19:52:49-cbafcc5/darwin-amd64-10_10 2018-10-03T19:54:16-9a8372f/darwin-amd64-10_10 2018-10-03T19:54:29-c91ce3c/darwin-386-10_11 2018-10-03T19:54:29-c91ce3c/darwin-amd64-10_10 2018-10-03T19:54:29-c91ce3c/darwin-amd64-race 2018-10-03T20:13:35-112f28d/darwin-amd64-10_10
ianlancetaylor
commented
5 years ago Any chance there was a recent change to whether the builders set GO_BUILDER_FLAKY_NET in the environment? Because the test does not fail in that way if that environment variable is set.
mikioh
commented
5 years ago Once http://golang.org/cl/102875 lands, I will smash the use of external DNS speakers such as rsc.io or some public full resolver then flaky tests disappear. For now, I have no spare time to debug the circumstances running the tests, also not sure there's a way to sneak in the (protected) circumstances.
andybons
commented
5 years ago @mikioh should we just disable this test until 1.13?
mikioh
commented
5 years ago Yes, please. Will try to allocate my spare time for Go 1.13 devel cycle.
gopherbot
commented
5 years ago Change https://golang.org/cl/151799 mentions this issue: net: skip flaky TestLookupDotsWithRemoteSource for now
andybons
commented
5 years ago @mikioh done. Assigning back to you. Thanks.
bcmills
commented
5 years ago Here's the same failure mode on dragonfly-amd64: https://build.golang.org/log/2f33b64e758b0fb89431e34f51f80580e5b56e59
--- FAIL: TestLookupDotsWithRemoteSource (10.85s)
lookup_test.go:669: LookupCNAME(www.mit.edu, mode=go): lookup www.mit.edu on 192.168.1.100:53: read udp 192.168.1.2:54367->192.168.1.100:53: i/o timeout
FAIL
FAIL net 68.224sfreebsd-arm-paulzhol (https://build.golang.org/log/daa40802f66a0d79d4d356d551c94456f56785b1):
--- FAIL: TestLookupDotsWithRemoteSource (12.52s)
lookup_test.go:656: LookupAddr(8.8.8.8): lookup 8.8.8.8.in-addr.arpa. on 192.168.1.253:53: read udp 192.168.1.101:38233->192.168.1.253:53: i/o timeout (mode=go)
FAIL
FAIL net 44.919s2020-01-25T05:48:12-01df660/freebsd-arm-paulzhol 2019-11-22T16:04:32-1c5bd34/freebsd-arm-paulzhol 2019-10-26T05:40:14-c3e8a20/linux-s390x-ibm 2019-09-18T19:24:59-90f9426/freebsd-arm-paulzhol 2019-08-19T21:42:54-d6ffc1d/freebsd-arm-paulzhol 2019-07-29T16:52:01-5235501/freebsd-arm-paulzhol 2019-07-01T14:45:49-bc593ea/freebsd-arm-paulzhol 2019-06-19T18:19:01-b373d31/freebsd-arm-paulzhol 2019-05-14T16:36:38-a9e107c/dragonfly-amd64 2019-05-09T18:02:19-c514071/dragonfly-amd64 2019-05-02T04:44:36-74b3c50/dragonfly-amd64 2019-04-28T23:51:04-50ad094/dragonfly-amd64
bcmills
commented
4 years ago I did some digging, and I believe that this flakiness is caused by the hard-coded timeout and attempt limits in net/dnsconfig_unix.go:
https://github.com/golang/go/blob/825ff1e3171a97fba1f29473d0be484ebfdc08b4/src/net/dnsconfig_unix.go#L41-L45
On my Linux workstation, man 5 resolv.conf reports these same defaults. However, given that the Go net.Resolver API accepts a context.Context which can provide its own timeout behavior, I think these hard-coded limits are out of place: they might make sense in the context of a legacy POSIX C API, but they don't really fit in a modern Go package.
I would recommend:
Complete this TODO to allow the Resolver to override the system-default timeout (and likely also the default limit on attempts).
Refactor the RemoteSource test to use a Resolver and explicit Context instead of the Context-less top-level LookupAddr function. Bound the timeout and number of retries to the test's deadline (#28135) instead of arbitrary constants.
bcmills
commented
2 years ago 
alexbrainman
commented
2 years ago Another flake by standard try-bot
https://storage.googleapis.com/go-build-log/43f55574/windows-386-2008_d562679e.log
while testing
https://go-review.googlesource.com/c/go/+/416975
Alex
mknyszek
commented
2 years ago Another flake:
gopherbot
commented
1 year ago Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource"Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource"Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource"Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource"Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource"Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource"Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource"Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource"Sorry, but there were parse errors in the watch flakes script. The script I found was:
#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource" && (`DNS server failure`|`no such host`|`server misbehaving`)And the problems were:
script:2.90: invalid syntax at |See https://go.dev/wiki/Watchflakes for details.
mateusz834
commented
8 months ago This test is failing again, now the 8.8.8.8.in-addr.arpa. seems to be broken.
[mateusz@arch ~ ]$ dig @8.8.8.8 8.8.8.8.in-addr.arpa PTR
; <<>> DiG 9.18.20 <<>> @8.8.8.8 8.8.8.8.in-addr.arpa PTR
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 47540
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; EDE: 0 (Other): ([209.244.0.1] Lame delegation at 8.8.8.in-addr.arpa for 8.8.8.8.in-addr.arpa/ptr)
; EDE: 0 (Other): ([209.244.0.2] Lame delegation at 8.8.8.in-addr.arpa for 8.8.8.8.in-addr.arpa/ptr)
; EDE: 22 (No Reachable Authority): (At delegation 8.8.8.in-addr.arpa for 8.8.8.8.in-addr.arpa/ptr)
;; QUESTION SECTION:
;8.8.8.8.in-addr.arpa. IN PTR
;; Query time: 53 msec
;; SERVER: 8.8.8.8#53(8.8.8.8) (UDP)
;; WHEN: Fri Dec 29 20:24:19 CET 2023
;; MSG SIZE rcvd: 288[mateusz@arch ~ ]$ dig @1.1.1.1 8.8.8.8.in-addr.arpa PTR
; <<>> DiG 9.18.20 <<>> @1.1.1.1 8.8.8.8.in-addr.arpa PTR
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41010
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;8.8.8.8.in-addr.arpa. IN PTR
;; Query time: 13 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Fri Dec 29 20:23:12 CET 2023
;; MSG SIZE rcvd: 49[mateusz@arch ~ ]$ dig @ns1.google.com 8.8.8.8.in-addr.arpa PTR
; <<>> DiG 9.18.20 <<>> @ns1.google.com 8.8.8.8.in-addr.arpa PTR
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45541
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;8.8.8.8.in-addr.arpa. IN PTR
;; ANSWER SECTION:
8.8.8.8.in-addr.arpa. 86400 IN PTR dns.google.
prattmic
commented
8 months ago https://github.com/golang/go/issues/64937#issuecomment-1874492864 notes that this seems to be fixed, though we're still seeing some failures (probably from cached responses?).
bcmills
commented
8 months ago I wouldn't say that this is “fixed” so much as “back in remission”.
Ideally the test should be made hermetic, but adding a hermetic DNS server to the Go repo for testing purposes seems like kind of a tough sell too. 😅
Neustradamus
commented
8 months ago It is an important ticket...
gopherbot
commented
7 months ago Found new dashboard test flakes for:
#!watchflakes
post <- pkg == "net" && test == "TestLookupDotsWithRemoteSource" && (`DNS server failure`||`no such host`||`server misbehaving`)
We are getting flaky test failures in net. This has been somewhat rare in the past few months, but it has failed the last 5 builds, so probably worth looking into now.
Looks very related to #16865 closed 2 years ago.