cannot run two times a script importing PyQt5

[aoloe]

#include <pybind11/embed.h>

namespace py = pybind11;

void first()
{
    py::scoped_interpreter guard{};
    auto scope = py::globals();

    py::exec(R"(
        import sys
        print(sys.argv)
        )", scope
    );

    py::exec(R"(
        import sys
        print(sys.argv)
        )", scope
    );

    py::exec(R"(
        from PyQt5.QtWidgets import QApplication
        print("first 1")
        )", scope
    );

    py::exec(R"(
        from PyQt5.QtWidgets import QApplication
        print("first 2")
        )", scope
    );
}

void second()
{
    py::scoped_interpreter guard{};
    auto scope = py::globals();

    py::exec(R"(
        import sys
        print(sys.argv)
        )", scope
    );
}

void third()
{
    py::scoped_interpreter guard{};
    auto scope = py::globals();

    py::exec(R"(
        from PyQt5.QtWidgets import QApplication
        print("third")
        )", scope
    );
}

int main()
{
    first();
    second();
    third();
}

when calling third() i get:

terminate called after throwing an instance of 'pybind11::error_already_set'
  what():  RuntimeError: PyQt5.QtWidgets cannot import type '����' from PyQt5.QtCore

At:
  <string>(3): <module>

Aborted

Am I doing something wrong or is it related to pybind/pybind11@326deef ?

[aoloe]

here a minimal code producing the error:

int main()
{
    {
        py::scoped_interpreter guard{};
        auto scope = py::globals();

        py::exec(R"(
            from PyQt5.QtWidgets import QApplication
            print("first")
            )", scope
        );
    }
    {
        py::scoped_interpreter guard{};
        auto scope = py::globals();

        py::exec(R"(
            from PyQt5.QtWidgets import QApplication
            print("second")
            )", scope
        );
    }
}

[aoloe]

when using PySide2 instead of PyQt5:

#include <pybind11/embed.h>

namespace py = pybind11;

int main()
{
    {
        py::scoped_interpreter guard{};
        auto scope = py::globals();

        py::exec(R"(
            from PySide2.QtWidgets import QApplication
            print("first")
            )", scope
        );
    }
}

correctly works but

#include <pybind11/embed.h>

namespace py = pybind11;

int main()
{
    {
        py::scoped_interpreter guard{};
        auto scope = py::globals();

        py::exec(R"(
            from PySide2.QtWidgets import QApplication
            print("first")
            )", scope
        );
    }
    {
        py::scoped_interpreter guard{};
        auto scope = py::globals();

        py::exec(R"(
            print("second")
            )", scope
        );
    }
}

gives me a "Segmentation Fault". I don't even need to load PySide2 in the second context.

[aoloe]

With PyQt5, this works:

#include <pybind11/embed.h>

namespace py = pybind11;

int main()
{
    {
        py::scoped_interpreter guard{};
        auto scope = py::globals();

        py::exec(R"(
            from PyQt5.QtWidgets import QApplication
            print("first")
            )", scope
        );
    }
    {
        py::scoped_interpreter guard{};
        auto scope = py::globals();

        py::exec(R"(
            print("second")
            )", scope
        );
    }
}

[dlong11]

@aoloe I am not sure this is a pybind11 issue. The same thing happens when you import numpy. https://github.com/numpy/numpy/issues/13051#issuecomment-510298962 My guess is this is a PyQt5 issue.

[aoloe]

@dlong11 , thanks for looking into this.

can you exclude that this is related to an effect similar 326deef ?

personally, i'm simply looking for a way to let the users launch python scripts (optionally with pyqt5) from inside of a c++ (qt) application... if somebody can suggest a different approach that does work, i will more than happy to try it out.

[dlong11]

@aoloe I can not exclude that this is related to 326deef

I am looking at the exact same thing on a project that I am working on, and noticed that if I initialize/import NumPy/finalize more than once I get a crash. This is due to the fact that NumPy has static variables that do not correctly get uninitialized/initialized correctly on unload/load.

I was guessing that PyQt5 might have a similar issue. (just guessing and could very well be wrong. 😄 ) With that being said if a user script imported NumPy or any other module that has static variables that don't get cleaned up correctly, they could run into the same issue.

Using plain python this will crash. (the docs do mention that some external modules will not work when doing multiple initialize/finalize (or py::scoped_interpreter)

Py_SetProgramName(Py_DecodeLocale("Test",0));
Py_Initialize();
PyImport_AddModule("__main__");
const char* content = "import os; print(os.__file__)\nimport numpy";
PyRun_SimpleString(content);
Py_Finalize();

Py_Initialize();
PyImport_AddModule("__main__");
const char* content = "import os; print(os.__file__)\nimport numpy";
PyRun_SimpleString(content);
Py_Finalize();

Thanks for linking to the other issue. I didn't see it. Even with it I am not banking on it fixing cases like the numpy case. Again I could be wrong.

I am also looking for a way to do this and would be happy if someone had any guidance around this.

[aoloe]

I did some further tests and this does work:

    {
        Py_SetProgramName(Py_DecodeLocale("Test",0));
        Py_Initialize();
        PyImport_AddModule("__main__");
        PyRun_SimpleString(sys_path.c_str());
        const char* content = "import os; print(os.__file__)\nimport PyQt5";
        PyRun_SimpleString(content);
        Py_Finalize();
    }

    {
        Py_Initialize();
        PyImport_AddModule("__main__");
        PyRun_SimpleString(sys_path.c_str());
        const char* content = "import os; print(os.__file__)\nimport PyQt5";
        PyRun_SimpleString(content);
        Py_Finalize();
    }

and this gives an error:

    {
        Py_SetProgramName(Py_DecodeLocale("Test",0));
        Py_Initialize();
        PyImport_AddModule("__main__");
        PyRun_SimpleString(sys_path.c_str());
        const char* content = "import os; print(os.__file__)\nfrom PyQt5.QtWidgets import QApplication";
        PyRun_SimpleString(content);
        Py_Finalize();
    }

    {
        Py_Initialize();
        PyImport_AddModule("__main__");
        PyRun_SimpleString(sys_path.c_str());
        const char* content = "import os; print(os.__file__)\nfrom PyQt5.QtWidgets import QApplication";
        PyRun_SimpleString(content);
        Py_Finalize();
    }

the output is:

/usr/lib/python3.7/os.py
/usr/lib/python3.7/os.py
Traceback (most recent call last):
  File "<string>", line 2, in <module>
RuntimeError: PyQt5.QtWidgets cannot import type '����' from PyQt5.QtCore

of, course running only the first half of the code does work correctly.

i will check with the pyqt5 guys, and see if they know about this.

[qaler]

Any updates? I'm encountering the same problem when I try to launch python scripts from different c++ threads (under different pybind11 scoped interpreter).

[aoloe]

@qaler could you please provide some minimal code to reproduce?

[Dariusz1989]

+1 from me as well, I also with to sure qt, precisely PySide2 with pybind11, I actually want to link it to my c++ Qt application so that it can properly interact with main c++ qt widget window.

[bstaletic]

Some libraries are just not safe to use after reinitializing the interpreter. As far as I know, the only options is to keep the interpreter alive and use gil_scoped_release and gil_scoped_acquire.

[aoloe]

@bstaletic i'm not talking about some libraries out there but about pyqt and pyside2 (and others about numpy)...

but i'm willing to explore other ways...
would you be willing to expand your thoughts and provide a version of the code above that works with your suggestions?
and possibly mention the steps one should take to make sure that as little resources as possible are kept alive when not in use...

i would be delighted to find something that works for my scripting engine!

[wjakob]

This is really not a pybind11-specific problem (your example snippets are pure CPython API), so I don't think it is helpful to discuss your ticket here, closing.

[aoloe]

@wjakob i understand that you want to see this ticket closes after so many time without action.

but this is one of the main issues that stops me from using pybind11 for creating a scripter engine that can use pyqt/pyside dialogs.

is there any place where i can start a discussion about this, discussion that will eventually produce a working solution (best practices) for using pybind11 with libraries that are not safe?

or does such a document already exist?

[bstaletic]

Like I said, many libraries are not safe to use after reinitializing the interpreter. NumPy is definitely one of them. PyQt apparently as well. Find a way to keep the interpreter alive. The following should work.

#include <pybind11/embed.h>

namespace py = pybind11;

int main()
{
    py::scoped_interpreter guard{};
    {
        auto scope = py::globals();
        py::exec(R"(
            from PyQt5.QtWidgets import QApplication
            print("first")
            )", scope
        );
    }
    {
        auto scope = py::globals();

        py::exec(R"(
            print("second")
            )", scope
        );
    }
}

[YannickJadoul]

but this is one of the main issues that stops me from using pybind11

Still doesn't mean it's a pybind11 issue. As @wjakob noted, this is CPython behavior, since you don't even use pybind11 in one of your examples. As @bstaletic suggested: don't initialize PyQT5 twice; maybe use subprocesses or something?

[aoloe]

i got pinged by bitbucket:

https://bitbucket.org/branchstudios/am_platform_external_python_pybind11/commits/326deef2ae6c535c2b1838fd7404579b3fe497b7

personally, i cannot judge if the commit is indeed a (good) solution i was facing, but i link it here so that better skilled people might be able to review it. (no idea either if the original author already planned to submit a patch upstream... sorry, if this comment is a duplicate or otherwise useless)

[aoloe]

a few years later, i've finally had a reason to try how things are with pyside6.

and from a first try, it seems that this issue is now solved

#include <pybind11/embed.h>

namespace py = pybind11;

int main()
{
    {
        py::scoped_interpreter guard{};
        auto scope = py::globals();

        py::exec(R"(
            from PySide6.QtWidgets import QApplication
            print("first")
            )", scope
        );
    }
    {
        py::scoped_interpreter guard{};
        auto scope = py::globals();

        py::exec(R"(
            from PySide6.QtWidgets import QApplication
            print("second")
            )", scope
        );
    }
}

does compile, run without and prints

first
second

i don't have pyqt5 installed on this machine and i cannot check / guess if the fix is on the side of pybind11 or because of pyside6, but with current versions everything seems to be working now.