[DocDB] Resume contentious waiter on tserver's rpc threadpool as opposed to the io thread pool

Description

We resume waiters from the wait-queue serially. In the process, When resuming a waiter, we try to require the shared in-memory locks with a deadline of 1s. And if this fails, we schedule the resumption of the scheduler which then tries to acquire the shared in-memory locks with deadline set to the rpc requests's deadline itself.

We seem to be using the wrong thread for resumption of such contentious waiters (the ones that couldn't acquire the shared in-memory locks in 1s). In particular, we are currently doing

          messenger_->scheduler().Schedule([serial_waiter](const Status& s) {
            if (!s.ok()) {
              serial_waiter.waiter->InvokeCallbackOrWarn(
                  s.CloneAndPrepend("Failed scheduling contentious waiter resumption: "));
              return;
            }
            ResumeContentiousWaiter(serial_waiter);
          }, std::chrono::milliseconds(100));

which uses

  void HandleTimer(const boost::system::error_code& ec) {
    ...
    auto now = std::chrono::steady_clock::now();
    while (!tasks_.empty() && (*tasks_.begin())->time() <= now) {
      io_service_.post([task = *tasks_.begin()] { task->Run(Status::OK()); });
      ...
    }
    ...
  }

and the ioservice points to a threadpool of size 4.

Messenger::Messenger(const MessengerBuilder &bld)
    : name_(bld.name_),
      ...
      io_thread_pool_(name_, FLAGS_io_thread_pool_size), // defaults to 4

We should have instead used Messenger's normal_thread_pool_ (size of 1024) which is used to execute all rpc requests.

The problematic change was part of this commit. Prior to that, the contentious waiter requests would have just failed with message - unable to acquire locks.

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

[X] I confirm this issue does not contain any sensitive information.

yugabyte / yugabyte-db