runtime: SIGSEGV in preemptone (riscv64)

[gopherbot]

#!watchflakes
default <- goarch == "riscv64" && builder == "linux-riscv64-mengzhuo" && `sigcode=1 addr=0xc0`  

Original flakes

#!watchflakes
default <- pkg == "golang.org/x/tools/internal/imports" && test == "TestModReplace2"

Issue created automatically to collect these failures.

Example (log):

=== RUN   TestModReplace2
SIGSEGV: segmentation violation
PC=0x54520 m=12 sigcode=1 addr=0xc0

goroutine 0 gp=0x3f504421c0 m=12 mp=0x3f50526708 [idle]:
runtime.preemptone(0x3f504421c0?)
    /home/swarming/.swarming/w/ir/x/w/goroot/src/runtime/proc.go:6297 +0x38 fp=0x3f5043ff28 sp=0x3f5043ff10 pc=0x54520
runtime.preemptall()
    /home/swarming/.swarming/w/ir/x/w/goroot/src/runtime/proc.go:6275 +0x60 fp=0x3f5043ff50 sp=0x3f5043ff28 pc=0x544c0
runtime.forEachPInternal(0x2fa878)
...
a3  0x223abf02  a4  0x3f98bb8000
a5  0x31a7d99   a6  0x29ab75fd
a7  0x1187f0    s2  0x3f5043fed0
s3  0x3f50526708    s4  0x3f50474000
s5  0x3f50241500    s6  0xffffffff
s7  0x4 s8  0x3f50038688
s9  0x3f5043fdc8    s10 0x2fa878
s11 0x3f504421c0    t3  0x2eb2a46908caf
t4  0xffffffffffffffff  t5  0x1913e15049b3
t6  0x3f50038408    pc  0x54520

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
default <- pkg == "golang.org/x/tools/internal/imports" && test == "TestModReplace2"

2024-08-13 17:00 x_tools-go1.23-linux-riscv64 tools@c1241b9c release-branch.go1.23@6885bad7 x/tools/internal/imports.TestModReplace2 [ABORT] (log)

=== RUN TestModReplace2 SIGSEGV: segmentation violation PC=0x54520 m=12 sigcode=1 addr=0xc0 goroutine 0 gp=0x3f504421c0 m=12 mp=0x3f50526708 [idle]: runtime.preemptone(0x3f504421c0?) /home/swarming/.swarming/w/ir/x/w/goroot/src/runtime/proc.go:6297 +0x38 fp=0x3f5043ff28 sp=0x3f5043ff10 pc=0x54520 runtime.preemptall() /home/swarming/.swarming/w/ir/x/w/goroot/src/runtime/proc.go:6275 +0x60 fp=0x3f5043ff50 sp=0x3f5043ff28 pc=0x544c0 runtime.forEachPInternal(0x2fa878) ... a3 0x223abf02 a4 0x3f98bb8000 a5 0x31a7d99 a6 0x29ab75fd a7 0x1187f0 s2 0x3f5043fed0 s3 0x3f50526708 s4 0x3f50474000 s5 0x3f50241500 s6 0xffffffff s7 0x4 s8 0x3f50038688 s9 0x3f5043fdc8 s10 0x2fa878 s11 0x3f504421c0 t3 0x2eb2a46908caf t4 0xffffffffffffffff t5 0x1913e15049b3 t6 0x3f50038408 pc 0x54520

— watchflakes

[mknyszek]

CC @golang/riscv64

[mengzhuo]

Updates: I've closed flake issues that with query "addr=0xc0 riscv64". All these failures related to same bad builder: linux-riscv64-mengzhuo--cm2

I found interesting logs in dmesg of this builder:

[12901.072321] INFO: task gc-stress:730941 blocked for more than 614 seconds.
[12901.079356]       Not tainted 6.6.36 #2.0~rc3.2+20240815152052
[12901.085355] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[12901.100078] task:gc-stress       state:D stack:0     pid:730941 ppid:730855 flags:0x00000004
[12901.100120] Call Trace:
[12901.100126] [<ffffffff810c173a>] __schedule+0x28c/0x848
[12901.100153] [<ffffffff810c1d3e>] schedule+0x48/0xd2
[12901.100161] [<ffffffff810c20c0>] schedule_preempt_disabled+0x16/0x28
[12901.100170] [<ffffffff810c4f96>] rwsem_down_write_slowpath+0x220/0x58e
[12901.100183] [<ffffffff810c5374>] down_write+0x70/0x72
[12901.100191] [<ffffffff801a8ae8>] vma_expand+0x46/0x1ca
[12901.100202] [<ffffffff801ac086>] mmap_region+0x3c0/0x6b0
[12901.100212] [<ffffffff801ac596>] do_mmap+0x220/0x39e
[12901.100219] [<ffffffff8018649a>] vm_mmap_pgoff+0x8c/0x118
[12901.100232] [<ffffffff801a968a>] ksys_mmap_pgoff+0x3a/0x158
[12901.100240] [<ffffffff80005562>] __riscv_sys_mmap+0x2a/0x36
[12901.100250] [<ffffffff810bf224>] do_trap_ecall_u+0xba/0x12e
[12901.100259] [<ffffffff810c8722>] ret_from_exception+0x0/0x6e
[12901.100273] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings

stack trace log below shows all these failure related to mmap call.

Bad builder is 4G RAM version of bananapi-f3, so I've update sysctl with

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

Note: This builder still able to respond to prometheus-node-exporter, so I didn't get any warning :(

[mengzhuo]

Updates 28th, Aug: I've made contact with SpacemiT staff who confirmed that hardware litmus test works as expected after a two days run.

I've also upgraded two builders with a backported kernel patch related to hang mmap scheduler https://lore.kernel.org/all/20231213203001.179237-5-alexghiti@rivosinc.com/

Unfortunately, it doesn't work.

Now, I've suspend these two builders and wait for a kernel fix from SpacemiT.