Assertion Error in threading module when finalize_interpreter

[abidrahmank]

Reproducible example code

Standalone setup is provided here : https://gist.github.com/abidrahmank/1ec549e13f600cd11eb70561c9223210

Just call make

Issue description

I have following workflow :

• I have a C library (crashloader.cpp) which will load other shared library plugins (threadcrash.cpp) and execute functions in that.

• Plugin libraries typically contains a function that creates an object (goo), execute it (execgoo) and delete it (delgoo).

• In my case, plugin embeds a Python script and execute it. So I initialize interpreter in object constructor and finalize it in destructor.

Issue : Everything works fine if I call exec in the same thread as creating the object. But the moment I call exec in a separate thread (which is my actual use case), I get following Assertion Error.

Exception ignored in: <module 'threading' from '......../lib/python3.6/threading.py'>
Traceback (most recent call last):
  File "......./lib/python3.6/threading.py", line 1299, in _shutdown
    assert tlock.locked()
AssertionError:  

[abidrahmank]

More data points

1. Assertion Error happens only when threading module is imported (either directly or indirectly via numpy import etc.)

2. Related code from threading

   def _shutdown():
   # Obscure:  other threads may be waiting to join _main_thread.  That's
   # dubious, but some code does it.  We can't wait for C code to release
   # the main thread's tstate_lock - that won't happen until the interpreter
   # is nearly dead.  So we release it here.  Note that just calling _stop()
   # isn't enough:  other threads may already be waiting on _tstate_lock.
   tlock = _main_thread._tstate_lock
   # The main thread isn't finished yet, so its thread state lock can't have
   # been released.
   assert tlock is not None
   assert tlock.locked()
   tlock.release()
   _main_thread._stop()
   t = _pickSomeNonDaemonThread()
   while t:
       t.join()
       t = _pickSomeNonDaemonThread()
   _main_thread._delete()

[abidrahmank]

One more data point

1. After further testing, I noticed that, if I import threading module immediately after py::initialize_interpreter in threadcrash.cpp, then it works fine.

/*updated threadcrash.cpp line:20*/
pybind11::initialize_interpreter()
py::module::import("threading");
// This also works : py::exec("import threading; print(threading._main_thread)");

[msardonini]

I also have this problem. Here is a quick way to reproduce it.

#include <thread>

#include "pybind11/embed.h"

namespace py = pybind11;

void test_func() {
  py::gil_scoped_acquire acquire;
  auto module = py::module_::import("numpy");  // Throws exception
  // auto module = py::module_::import("sys");  // does not throw expction
  // auto module = py::module_::import("threading");  // throws
}

int main() {
  py::scoped_interpreter guard;  // Python exception thrown in destructor of 'guard'
  py::gil_scoped_release release;
  {
    std::thread my_thread = std::thread(test_func);
    my_thread.join();
  }
}

I see the output:

Exception ignored in: <module 'threading' from '/app/.pyenv/versions/3.8.10/lib/python3.8/threading.py'>
Traceback (most recent call last):
  File "/app/.pyenv/versions/3.8.10/lib/python3.8/threading.py", line 1373, in _shutdown
    assert tlock.locked()
AssertionError:

Is there something we should be doing at the end of the thread before module is destroyed? Can someone provide some context on what the problem is here?

Any help is appreciated,

[moratom]

@msardonini did you by any chance find any solutions for this? I'm also running into the same issue.