Open gopherbot opened 6 months ago
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
The most recent ppc64le power9 failures indicate OOM. The ppc64le containers are limited to 10GB today (including tmpfs for LUCI stuff). The rangegen test is probably the most memory intensive test.
Memory cgroup out of memory: Killed process 2631390 (compile) total-vm:3672064kB, anon-rss:2462528kB, file-rss:128kB, shmem-rss:16704kB, UID:2000 pgtables:350kB oom_score_adj:0
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
I've readjusted the power9 builders to 4 instances, and reallocated the memory (about 3G more to the remaining buildlets). I suspect the underlying hardware is overcommitted, and that is the cause of the spurious slowdowns we see only on P9.
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "cmd/internal/testdir" && test == "Test/rangegen.go"
Issue created automatically to collect these failures.
Example (log):
— watchflakes