Closed akoshelev closed 2 months ago
Draft run for 50M records (was timing out before): https://draft-mpc.vercel.app/query/view/paved-stork-5287
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 91.50%. Comparing base (
195ba00
) to head (4215af5
). Report is 14 commits behind head on main.:exclamation: Current head 4215af5 differs from pull request most recent head 76a2b27
Please upload reports for the commit 76a2b27 to get more accurate results.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
it does not hang anymore, fails because of #1100
I decided not to create an executor module for now - block_in_place
is a specialized API and maybe we use some other runtime sooner than we run into another issue. If block_in_place
needs to be used somewhere else, then a new module will be justified
This is an attempt to mitigate #1120 by hinting Tokio executor that query task maybe blocking the worker thread for longer than expected and it should hand off everything from that worker thread.
Here is how this issue is reproducible in our environment
Tokio executor is a work-stealing scheduler that optimizes for low latency and resource utilization. It optimizes for low scheduling overhead and uses the same worker threads for polling tasks and to service IO and timer operations, if they are enabled. This leads to very low overhead, but also to an unfortunate possibility of tasks that are long-running to block the entire runtime, even if it is multithreaded. Here is an example in IPA code that led to this discovery:
Client asks for IPA query status and times out, because IPA application is not responding.
Ultimately tokio team decided that it is not a bug in the implementation and made it clear that application needs to take care of it. There are a few ways to deal with it, one of them is to signal the executor that this task is about to block the worker thread for long time, so it has a chance to move the IO driver and pending tasks to other threads. This is what this proposed solution does.