Open jrray opened 1 year ago
Hi @jrray, I think this should be caused by the AsyncWaitGroup
implementation of https://github.com/al8n/wg. I suspect we need to replace parking_lot::Mutex
and parking_lot::RwLock
with async locks for the AsyncCache
.
We have encountered a deadlock situation when attempting to have nested caches, as in a key-key-value multi-dimensional map, where there are many spawned tasks trying to interact with the cache at once, and the tasks attempt to call
wait
after insert of the inner value.All the tokio (non-blocking) threads become blocked on trying to obtain a lock on the cache. The
await
insidewait
allows one of the other tasks destined to block to be scheduled, eventually causing deadlock. The code below,inner_cache.wait()
happens while a read lock onlru
is held. Avoiding theinner_cache.wait()
call prevents the deadlock.We have restructured our code so it no longer uses nested caches and avoids this situation, but I wanted to share this discovery. This may not be a usage pattern that is intended to work or be supported. But to my eyes it is not immediately obvious that this would deadlock. I speculate that if the
CacheProcessor
was spawned as a dedicated thread instead of a green thread it might be able to complete the wait group and unstick things (we didn't experience a deadlock with this same design pattern when using the syncCache
).Here is an example test that can demonstrate the deadlock: