Open gopherbot opened 1 month ago
Found new dashboard test flakes for:
#!watchflakes
default <- pkg == "runtime/pprof" && test == "TestProfilerStackDepth/mutex"
I spent some time looking into this failure, and it's clearly some kind of GC deadlock. Notably, the goroutine trying to write out the mutex profile calls into malloc, then into GC assists, and gets stuck somewhere in assists. Unfortunately it's unclear where. Some candidates include:
The lack of any smoking guns in the stack dump suggests to me that sysmon
may be involved. Or, it's some kind of memory corruption. I also wonder if this is a platform-specific failure; windows-386 is a bit funky.
A final note is all the GC mark workers in the stack dump are parked. This kinda makes sense -- the GC is not done, but they can't find any work to do because in 2 minutes they almost certainly found everything they could aside from wherever that one goroutine is stuck.
Actually, the assist queue lock theory is not possible. We know for a fact we're in gcAssistAlloc1
since we're on the system stack and the line numbers match up for us to have entered the system stack there.
So, the likely explanation is difficulty preempting some goroutine (I can't think of any other reason we'd block so long...).
I have a new possible lead/theory. This test is somewhat special in that the mutex profile rate is 1. What if there's a place where we're acquiring and/or releasing a runtime lock, somewhere deep in gcDrainN
, where we end up recording that lock in the mutex profile and blocking somewhere in there. I don't immediately see why that would cause a deadlock, but since this is a new kind of failure, recent changes (such as in the Go 1.23 cycle) may be relevant.
Though, this doesn't really line up with the evidence, since there should be some other goroutine waiting on a lock. Maybe it's a self deadlock? The write-mutex-profile code is interacting with the same locks that the runtime unlock path is, though I don't immediately see how we could self-deadlock in exactly the place we're stuck. Even if we were holding a lock (which it doesn't look like we are), that should prevent recording further locks.
In the last triage meeting, I said we'd just wait for more failures. So, setting the WaitingForInfo label.
Issue created automatically to collect these failures.
Example (log):
— watchflakes