Open eranrund opened 3 months ago
This might be relevant: https://www.reddit.com/r/rust/comments/urnqz8/different_behaviors_of_recursive_read_locks_in/
Basically, the write lock has a higher priority than the read lock. When trying to acquire a write lock in between two recursive read locks, the second read lock cannot be acquired, which in turn prevents the release of the first read lock.
Looks like this is even in the official docs: https://doc.rust-lang.org/std/sync/struct.RwLock.html
So I bet this is whats happening here
Hello,
We've run into a deadlock in mobilecoind twice now and wondering if you also happened to run into it. The way we have observed this is that
mobilecoind
stops responding to GRPC calls. The two times we encountered it we attachedgdb
and were able to learn that the GRPC worker thread (of which there is only one appears to be blocked on locking an rwlock.Here's the list of threads at the time this happened:
And the stacktrace of each thread:
Pay attention to thread 23 -
23 Thread 0x7f2c682be700 (LWP 61) "Mobilecoind-RPC"
:If this stacktrace is indeed correct, this is pretty weird:
get_network_status()
first locksself.network_state
here. It then callsget_last_block_infos
here which tries to lock the same lock: https://github.com/mobilecoinfoundation/mobilecoin/blob/main/mobilecoind/src/service.rs#L230Assuming compiler optimization didn't do anything funky here, it is pretty surprising that the first lock succeeded and the second one got stuck.
I couldn't find anything else in the other threads stack traces that could shed some light on it. The
self.network_state
lock is locked for writing in only one place as far as I can tell: https://github.com/mobilecoinfoundation/mobilecoin/blob/main/ledger/sync/src/ledger_sync/ledger_sync_service_thread.rs#The stack trace of the ledger sync thread is not very useful: