Closed dmik closed 3 years ago
DosWaitEventSem
is called in __fmutex_request_internal
and it supports ignoring ERROR_INTERRUPT via a _FMR_IGNINT flag (which is used in case of _um_heap_lock
) by retrying the wait cycle. When the flag is present, ERROR_INTERRUPT is replaced with ERROR_TIMEOUT which in turn is the only condition to retry the loop (if the wait succeeds or fails with any other error, the cycle is aborted and the result is returned to the caller). ERROR_TIMEOUT is there because I guess that the wait cycle was originally designed to spin in 3 second wait intervals in order to implement owner death (and a subsequent deadlock) detection.
The owner check begins with checking if the mutex is owned (if its owner field is not 0) in the end of the wait loop. The code obviously assumes that if there is no owner but DosWaitEventSem
returns ERROR_TIMEOUT, something went really wrong and bails out with a non-zero error code (ERROR_TIMEOUT) indicating that ownership was not transferred. In our case it's most likely ERROR_INTERRUPT which causes this behavior but nevertheless a failure is returned and _fmutex_abort
gets triggered.
The actual problem here is that the assumption the code makes about the owner is generally not right. If the owner is zero after ERROR_TIMEOUT (and ERROR_INTERRUPT), it doesn't necessarily means that something went wrong. It could be that the mutex owner simply released the mutex (and reset the owner field to 0) the next moment (time slice) after DosWaitEventSem
returned those errors. I.e. the mutex request code should simply gain ownership (set itself as the mutex owner) and return success.
The reason why it could not be seen earlier is because it's just really unlikely that the above scenario will happen exactly after 3 seconds of unsuccessful waiting (where ERROR_TIMEOUT is returned). However, breaking the wait with ERROR_INTERRUPT may happen much more frequently than every 3 seconds in Chromium (every few milliseconds at some stages!) which makes this scenario much more likely.
I will try to fix the code and see if the error is gone.
Note that using _fmutex
in pthread (see https://github.com/bitwiseworks/pthread-os2/issues/9) made this problem even more frequent. And the solution I'm going to commit makes it go away according to my testing.
Sometimes with multi-process mode enabled in Chromium (https://github.com/bitwiseworks/qtwebengine-chromium-os2/issues/12), I get this from one of the processes:
The stack trace is as this:
The "LIBC Heap" mutex is used to serialize access to the LIBC heap. From what I can tell,
_um_heap_lock (h)
is called in_umalloc
which in turn calls_fmutex_checked_request (&h->fsem, _FMR_IGNINT)
which calls_fmutex_request
with the same arguments which returns non-zero (an error) which causes_fmutex_abort
to be called.I don't have a 100% proof (since that I don't have full LIBC logging in this run) but it really smells like
DosWaitEventSem
returningERROR_INTERRUPT
while performing a LIBCx handle transfer (see bitwiseworks/libcx#87).