Passing python lambdas/functions to C++ std::thread not working

[voertler]

Hello,

currently I'm trying to use cppyy in an application that uses C++ std::threads. When passing a Python lambda function to a C++ thread I got segmentation faults or a blocking behavior.

This is a stripped down example:

import cppyy
import os

def define_threading():
    cppyy.cppdef("""
    #include <functional>
    void launch_thread(std::function<void()> lambda) {
    // Calling lambda() outside of thread works
    // lambda();
    std::thread thread([&lambda]() {      
         lambda();
    });
    thread.join();
    }
    """)

def my_function() -> 'void':
    print("Hello from Python lambda")

if __name__ == "__main__":
    define_threading()
    #C++ lambda works
    #cppyy.cppexec("""
    # launch_thread([]() 
    #     {
    #         std::cout << "Hello from C++ lambda" << std::endl;
    #     }
    # );
    # """) 
    cppyy.gbl.launch_thread(my_function)

There are several observations:

• When executing my_function in the context of a C++ std::thread the python interpreter hangs and the python process needs to be killed
• When passing a C++ lambda (commented out code) it can be executed within the std::thread
• Calling the python lambda outside the thread works

I used cppyy 3.2.1 both on an Ubuntu 24.04 installed through pip with c++2a enabled, as well as a custom built version on Rocky8 with c++17. Both versions behave in the same way.

[wlav]

Python has a Global Interpreter Lock which needs to be held for executing any Python code. Most Python-C++ binders release the GIL by default when entering C++ under the assumption that there's no Python being run in C++. Cppyy, however, does not: most Python code is single-threaded and releasing/re-acquiring the GIL isn't cheap. Furthermore, philosophically, I fall in the camp that thinks that threaded programming is better off with a non-shared default behavior.

So in this case, launch_thread holds the GIL, making it impossible for the callback to my_function to acquire it. To have this code run, do:

    cppyy.gbl.launch_thread.__release_gil__ = True
    cppyy.gbl.launch_thread(my_function)

(Although technically, you must also wait in the main thread for completion of the child.)

[voertler]

Thanks a lot, this helps a and I made progress on an more complex library integration. However, calling __release_gil__ for each function which might use a thread internally is cumbersome. Is there an easy way to disable the GIL for all functions?

[wlav]

Global changes seem like a bad idea, as it will affect all code, not just yours. If all your code is in a namespace, you can write a pythonization that is namespace-specific and simply flips all __release_gil__ of methods that fly by. Doesn't work for free functions, though. Example:

import cppyy
import cppyy.types

cppyy.cppdef("""\
namespace AAA {
class BBB {
public:
    void foo() {}
}; }""")

def release_gil(klass, name):
    for attr in klass.__dict__.values():
        if isinstance(attr, cppyy.types.Method):
            attr.__release_gil__ = True

cppyy.py.add_pythonization(release_gil, "AAA")

BBB = cppyy.gbl.AAA.BBB
print(BBB.foo.__release_gil__)

[voertler]

Thaks a lot, this gives exactly the control that I need.