Possible deadlock under high load, due to fixed-capacity channels?

thomcc commented 2 years ago

https://github.com/ockam-network/ockam/blob/8bf1502c8a105c874e8d493097c6c57948183a09/implementations/rust/ockam/ockam_node/src/context.rs#L129 (and some others) use a fixed-capacity channel (which if full, blocks the sender until it isn't).

This is surprising to me since there's a pretty classic deadlock that can happen because of these in some use cases like ours... Or at least it seems like it could — there's enough going on in ockam_node that I can't be sure if this is handled — it could be.

Specifically, the situation where I worry a deadlock happens is:

If you have two workers, W1 and W2, where W1 and W2 which may message each-other at least some of the time. (I think they need to be local workers too, but maybe I'm just lacking imagination).
Both W1 and W2 are very nearly at their channels capacity. (For example, if they're unable to process messages fast enough, and the backlog fills all the way up.
Now, W1 sends a message to W2. Because W2s channel is full, the send(msg).await wont resolve until W2 pops the next item of its channel and W1's message finds space.
Before W2 gets to do that (while W1 is still blocked), W2 sends W1 a message.
Sadly, W1's channel is also at capacity, so W2's send(msg).await won't wont resolve until W1 pops the next item of its channel.

That is — W1 is blocked until W2 pops an item off its queue... and W2 is also blocked until W1 pops an item off its queue.

That said, I don't know for sure this is actually possible — message sending in ockam is involved enough that it might not be, and might just look possible for ockam. (Concretely, I'm assuming ctx.send_message bottoms out on something like a channel::send to these channels)

That said, I have see this type of deadlock before in other projeces, so even though it sounds unlikely, it definitely happens under high load (especially if you arent on full release build).