Hand off Tokio bookkeeping tasks before starting a new IPA query

[akoshelev]

This is an attempt to mitigate #1120 by hinting Tokio executor that query task maybe blocking the worker thread for longer than expected and it should hand off everything from that worker thread.

Here is how this issue is reproducible in our environment

Tokio executor is a work-stealing scheduler that optimizes for low latency and resource utilization. It optimizes for low scheduling overhead and uses the same worker threads for polling tasks and to service IO and timer operations, if they are enabled. This leads to very low overhead, but also to an unfortunate possibility of tasks that are long-running to block the entire runtime, even if it is multithreaded. Here is an example in IPA code that led to this discovery:

• IPA query kicks in and first step it performs a very large reallocation, keeping the task running for minutes before yielding.
• At the time when query started, there was only one active thread in the runtime, and it began polling this task
• because other threads are parked, there is nothing that can indicate to the executor that there are more tasks that are ready. Tokio Runtime is not polling any task.

• Client asks for IPA query status and times out, because IPA application is not responding.

  Ultimately tokio team decided that it is not a bug in the implementation and made it clear that application needs to take care of it. There are a few ways to deal with it, one of them is to signal the executor that this task is about to block the worker thread for long time, so it has a chance to move the IO driver and pending tasks to other threads. This is what this proposed solution does.

[akoshelev]

Draft run for 50M records (was timing out before): https://draft-mpc.vercel.app/query/view/paved-stork-5287

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 91.50%. Comparing base (195ba00) to head (4215af5). Report is 14 commits behind head on main.

:exclamation: Current head 4215af5 differs from pull request most recent head 76a2b27

Please upload reports for the commit 76a2b27 to get more accurate results.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1123 +/- ## ========================================== + Coverage 91.38% 91.50% +0.11% ========================================== Files 188 189 +1 Lines 27066 27335 +269 ========================================== + Hits 24735 25012 +277 + Misses 2331 2323 -8 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[akoshelev]

it does not hang anymore, fails because of #1100

[akoshelev]

I decided not to create an executor module for now - block_in_place is a specialized API and maybe we use some other runtime sooner than we run into another issue. If block_in_place needs to be used somewhere else, then a new module will be justified