Currently the Mutex is relying on setInterval(keepAlive, 5000) to refresh it's lease on the mutex. However, this is fatally flawed: there is no guarantee cb will run every 5 seconds. It may not even ever run at all!
In production, we're encountering a scenario where synchronous Markdown parser is running for ~90 seconds. Since JavaScript is single threaded, that means the Mutex is not renewed in time, and other threads decide our thread has crashed, so they go ahead and take the Mutex. Then 85 seconds later, our code continues on, thinking it still has the mutex, then it releases it, only to discover either a) it has lost ownership or b) it is double-freed (because it lost ownership but then the other thread finished its stuff and released the mutex, so nobody has it now).
The only reliable way to fix this is to always double check that our mutex lock hasn't expired before we do things that require the lock. If it has expired, we try to renew our lock. If that fails we throw an ETIMEOUT.
Currently the Mutex is relying on
setInterval(keepAlive, 5000)
to refresh it's lease on the mutex. However, this is fatally flawed: there is no guaranteecb
will run every 5 seconds. It may not even ever run at all!In production, we're encountering a scenario where synchronous Markdown parser is running for ~90 seconds. Since JavaScript is single threaded, that means the Mutex is not renewed in time, and other threads decide our thread has crashed, so they go ahead and take the Mutex. Then 85 seconds later, our code continues on, thinking it still has the mutex, then it releases it, only to discover either a) it has lost ownership or b) it is double-freed (because it lost ownership but then the other thread finished its stuff and released the mutex, so nobody has it now).
The only reliable way to fix this is to always double check that our mutex lock hasn't expired before we do things that require the lock. If it has expired, we try to renew our lock. If that fails we throw an
ETIMEOUT
.