[BUG]: gil_scoped_acquire blocks

onelxj commented 1 year ago

Required prerequisites

[X] Make sure you've read the documentation. Your issue may be addressed there.
[X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
[X] Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

pybind11_bazel commit: fc56ce8a8b51e3dd941139d329b63ccfea1d304b

Problem description

I have a struct Middleware in C++ and I want to hold scoped_interpreter py::object and event_loop_thread as fields. In the method initialize I have some std::thread, that will invoke some python code there.

Since it is multithreaded, I want to acquire in the thread, but it just blocks and hangs, no error is thrown, the release before the thread start didn't help as well.

Any suggestions please?

No other code running.

struct Middleware{
  void initialize(){
    py::gil_scoped_release release; // didn't help ???
    event_loop_runner = std::thread([&]() {
      {
        {
          py::gil_scoped_acquire acquire; // ??? blocks for some reason
          py::object module = py::module::import("simple.python");
          event_loop_thread = module.attr("EventLoopThread")();
          event_loop_thread.attr("start")(); // non-blocking
        }
        std::unique_lock<std::mutex> lock(python_thread_lock_);
        cv.wait(lock, [&]() { return !is_running; }); // block until stop called.
      }
    });
  }

  pybind11::scoped_interpreter guard;
  py::object event_loop_thread;
   std::thread event_loop_runner;
};

Reproducible example code

No response

Is this a regression? Put the last known working version here if it is.

Not a regression

feltech commented 12 months ago

The one time I saw something like this it wasn't the GIL causing the hang per se, it was because we'd accidentally linked libpython into the extension module. Perhaps worth a check.

Ahajha commented 9 months ago

Hopefully you have a solution by now, but I think you might want something like this (untested):

struct Middleware{
  void initialize(){
    event_loop_runner = std::thread([&]() {
      {
        {
          py::gil_scoped_acquire acquire;
          py::object module = py::module::import("simple.python");
          event_loop_thread = module.attr("EventLoopThread")();
          event_loop_thread.attr("start")(); // non-blocking
        }
        std::unique_lock<std::mutex> lock(python_thread_lock_);
        cv.wait(lock, [&]() { return !is_running; }); // block until stop called.
      }
    });
  }

  pybind11::scoped_interpreter guard;
  py::gil_scoped_release release;
  py::object event_loop_thread;
  std::thread event_loop_runner;
};

On creation, the thread that the scoped interpreter is created in has the GIL. You then need to release it (which we're doing by automatically constructing a scoped release after the scoped interpreter is initialized), then it can be acquired in another thread.

Your previous approach would only release the GIL for the scope of creating the thread, which likely wouldn't affect it.

pybind / pybind11