Open abidrahmank opened 4 years ago
More data points
Assertion Error happens only when threading
module is imported (either directly or indirectly via numpy import etc.)
Related code from threading
def _shutdown():
# Obscure: other threads may be waiting to join _main_thread. That's
# dubious, but some code does it. We can't wait for C code to release
# the main thread's tstate_lock - that won't happen until the interpreter
# is nearly dead. So we release it here. Note that just calling _stop()
# isn't enough: other threads may already be waiting on _tstate_lock.
tlock = _main_thread._tstate_lock
# The main thread isn't finished yet, so its thread state lock can't have
# been released.
assert tlock is not None
assert tlock.locked()
tlock.release()
_main_thread._stop()
t = _pickSomeNonDaemonThread()
while t:
t.join()
t = _pickSomeNonDaemonThread()
_main_thread._delete()
One more data point
py::initialize_interpreter
in threadcrash.cpp, then it works fine./*updated threadcrash.cpp line:20*/
pybind11::initialize_interpreter()
py::module::import("threading");
// This also works : py::exec("import threading; print(threading._main_thread)");
I also have this problem. Here is a quick way to reproduce it.
#include <thread>
#include "pybind11/embed.h"
namespace py = pybind11;
void test_func() {
py::gil_scoped_acquire acquire;
auto module = py::module_::import("numpy"); // Throws exception
// auto module = py::module_::import("sys"); // does not throw expction
// auto module = py::module_::import("threading"); // throws
}
int main() {
py::scoped_interpreter guard; // Python exception thrown in destructor of 'guard'
py::gil_scoped_release release;
{
std::thread my_thread = std::thread(test_func);
my_thread.join();
}
}
I see the output:
Exception ignored in: <module 'threading' from '/app/.pyenv/versions/3.8.10/lib/python3.8/threading.py'>
Traceback (most recent call last):
File "/app/.pyenv/versions/3.8.10/lib/python3.8/threading.py", line 1373, in _shutdown
assert tlock.locked()
AssertionError:
Is there something we should be doing at the end of the thread before module
is destroyed? Can someone provide some context on what the problem is here?
Any help is appreciated,
@msardonini did you by any chance find any solutions for this? I'm also running into the same issue.
Reproducible example code
Standalone setup is provided here : https://gist.github.com/abidrahmank/1ec549e13f600cd11eb70561c9223210
Just call
make
Issue description
I have following workflow :
I have a C library (crashloader.cpp) which will load other shared library plugins (threadcrash.cpp) and execute functions in that.
Plugin libraries typically contains a function that creates an object (goo), execute it (execgoo) and delete it (delgoo).
In my case, plugin embeds a Python script and execute it. So I initialize interpreter in object constructor and finalize it in destructor.
Issue : Everything works fine if I call exec in the same thread as creating the object. But the moment I call exec in a separate thread (which is my actual use case), I get following Assertion Error.