In several cases, we avoid blocking the asyncio event loop by wrapping rp calls in ThreadPoolExecutor threads. However, even if the wrapping asyncio Tasks time out, a running Thread cannot generally be cancelled. If the rp call blocks forever, the thread pool could fill up.
We need to make sure that all wrapped rp calls have some sort of guarantee of completion, or we need to appropriately treat timeouts of thread futures as potentially critical internal disruptions of the program state.
In some cases, we can look for alternative coding constructs that allow for periodic polling or for an Exception to be injected that would interrupt the Thread.
Near-term steps in addressing this issue include enumerating the non-trivial occurrences of potentially-hanging wrapped blocking calls.
In several cases, we avoid blocking the asyncio event loop by wrapping
rpcalls in ThreadPoolExecutor threads. However, even if the wrapping asyncio Tasks time out, a running Thread cannot generally be cancelled. If therpcall blocks forever, the thread pool could fill up.We need to make sure that all wrapped rp calls have some sort of guarantee of completion, or we need to appropriately treat timeouts of thread futures as potentially critical internal disruptions of the program state.
In some cases, we can look for alternative coding constructs that allow for periodic polling or for an Exception to be injected that would interrupt the Thread.
Near-term steps in addressing this issue include enumerating the non-trivial occurrences of potentially-hanging wrapped blocking calls.