adding a pythran CLI flag to support free-threaded CPython

[rgommers]

For free-threaded CPython (xref PEP 703, https://py-free-threading.github.io/), Pythran seems to be working quite well already. We've tested Pythran-generated extension modules reasonably well within SciPy now with GIL-free multi-threaded tests, and so far everything looks healthy. There is one thing to do in Pythran though, namely adding a compile option to allow the user to request that the extension module is marked as compatible with free-threading.

That is pretty straightforward, it looks like this (see https://py-free-threading.github.io/porting/#declaring-free-threaded-support):

static PyModuleDef_Slot module_slots[] = {
    ...
#ifdef Py_GIL_DISABLED
    {Py_mod_gil, Py_MOD_GIL_NOT_USED},
#endif

or for single-phase initialization:

PyMODINIT_FUNC
PyInit__module(void)
{
...
#ifdef Py_GIL_DISABLED
    PyUnstable_Module_SetGIL(mod, Py_MOD_GIL_NOT_USED);
#endif

The way this is done in Cython and numpy.f2py is similar, with an opt-in flag to cython and f2py:

• Cython: cython -Xfreethreading_compatible=True (xref https://github.com/cython/cython/pull/6242)
• NumPy f2py: f2py --freethreading-compatible (xref https://github.com/numpy/numpy/pull/26981/files)

For Pythran I'd propose the same, an opt-in flag that when given inserts one of the #ifdef Py_GIL_DISABLED snippets in the generated C++ code:

pythran --freethreading-compatible

[serge-sans-paille]

Thanks for the extra pointers. This could be a config option, probably backend.freethreading_compatible=true and indeed, pythran only uses the C API for conversion so a good start would be to update the module initialization and start from there. I can handle that, unless you already have experience in that particular topic and want to give it a go?

[rgommers]

I can handle that, unless you already have experience in that particular topic and want to give it a go?

If you could handle it, that'd be great. The C module initialization part I did before, and it's the simplest part here probably - there is nothing more to it than the code snippets above. How to plumb it into Pythran and expose the CLI flag is most of the work I guess, and I'm not too sure about that.

The other part is perhaps if you'd want a CI job for free-threading? That is probably a matter of:

1. Adding cp313t to python-versions at https://github.com/serge-sans-paille/pythran/blob/master/.github/workflows/core.yml#L15
2. Changing actions/setup-python@v4 in that same job to quansight-labs/setup-python@v5.3.1 in that same file. For context: the official setup-python has not been responsive to the PR to add free-threading support for months, so we had to fork it (xref https://github.com/Quansight-Labs/free-threaded-compatibility/issues/107). The setup-python maintainers promised to do something about it "in the next quarter", so then we can switch back.

[serge-sans-paille]

perfect, I'll handle this