search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
124.37k stars 17.71k forks source link

x/build: netbsd-amd64-8_0 builder timing out frequently #31858

Open bcmills opened 5 years ago

bcmills commented 5 years ago

Four of the last five failures on the netbsd-amd64-8_0 builder are timeouts in various tests. I can't tell whether the builder is slow or the test is triggering some sort of deadlock.

https://build.golang.org/log/a64cd49d7d367f90e6b79ab84714b4c065763168 https://build.golang.org/log/f7f50b085698aa5480d2c34a3e6c250595317bf6 https://build.golang.org/log/7b749c8616a9b7e9a77c7735038673f9a9dc5d37 https://build.golang.org/log/26dd20510233ec659adf0d5987690164d8abaa6a

CC @bsiegert @bradfitz

bsiegert commented 5 years ago

I think it's just slowness. Maybe increasing the timeout helps?

bradfitz commented 5 years ago

@bsiegert, what makes you think that? I suspect the NetBSD port (or NetBSD) is buggy based on looking at historical data over the past month on this builder. The cgo_test has a 43.2 second median over 309 runes. And the cmd/compile/internal/types test is fast across all builders: 3.4 seconds on linux-amd64 at the slowest, 34.1 seconds on linux-arm on physical ARM hardware. js-wasm does it in 8.2 seconds. Why can't netbsd-amd64 do it in under 20 minutes?

So, no, I don't think it's just slowness.

zoulasc commented 5 years ago

Makes sense (that the problem lies with NetBSD or the NetBSD port). Is there a crash dump of where it is stuck?

bradfitz commented 5 years ago

@zoulasc, nope. Our build system just gives up on it and doesn't try to actively grab a stack. (which varies by OS, anyway) If you want to give me basic instructions for capturing some debug info we could probably wire it up to report that before failing.

zoulasc commented 5 years ago

I will setup a local build and try to reproduce it locally so I can give you something that works...

bcmills commented 5 years ago

Still an issue. misc/cgo/test, os/signal, and go/types all have recent timeouts: https://build.golang.org/log/a1652930ea41da5e9588f973a4c8a91878c00062 https://build.golang.org/log/9fcd7b26fe2a11d9883c6c18c370e36e3b3039b0 https://build.golang.org/log/a511a477b595ee1e3f444ed2b3e943bbc9e65541 https://build.golang.org/log/c0e5926e39b4704f6ff9bcfb23e41de1237ce859

There is also a cmd/go timeout that may or may not be related: https://build.golang.org/log/8438c6756c49694e123c5b058e2a7903ece9d7f1

gopherbot commented 5 years ago

Change https://golang.org/cl/185217 mentions this issue: env/netbsd-amd64: update to NetBSD 8.1

gopherbot commented 4 years ago

Change https://golang.org/cl/226977 mentions this issue: dashboard, env/netbsd-{amd64,386}: add NetBSD 9.0 builders