search

python-cffi / cffi

A Foreign Function Interface package for calling C libraries from Python.
https://cffi.readthedocs.io/en/latest/
Other
133 stars 42 forks source link

Can free-threading wheel builds be made available on PyPI? #119

Closed rtobar closed 2 months ago

rtobar commented 2 months ago

I know from other issues/PRs (e.g., https://github.com/python-cffi/cffi/issues/40) that making cffi actually thread-safe is a big task. However, a small but easier step is to produce free-threaded wheels. As mentioned in https://docs.python.org/3.13/howto/free-threading-extensions.html#module-initialization

Extension modules need to explicitly indicate that they support running with the GIL disabled; otherwise importing the extension will raise a warning and enable the GIL at runtime.

Thus, providing FT wheels doesn't necessarily mean the extension supports it, it simply allows the package to be installed in a FT installation.

I do understand that providing such wheels could make some users believe that cffi is fully multi-threaded. As long as expectations are made clear that hopefully shouldn't be an issue, but that's very easy to say.

In my case I'm trying to provide FT builds of my library (with hopefully actual thread-safe code), and cffi is an optional dependency (rarely used though from what I know). I'm not running multi-threaded code in my tests (yet), and even if I did python would should re-enable the GIL when importing cffi.

arigo commented 2 months ago

As far as I can tell, CFFI will crash Python or corrupt memory if you force it to run in a free-threaded Python. That's why we don't provide free-threaded CFFIs.

nitzmahone commented 2 months ago

There are numerous issues to solve before CFFI has any real hope of "just working" under the t free-threaded ABI- even with the auto-GIL fallback. Things have come a long way since the last time I tried it around beta4- I poke at things every few months, since I want this to work just as badly as a lot of other folks... IIUC the biggest (but not only) blockers right now to getting it to work even with the GIL fallback are around the lack of limited/stable API/ABI support in the free-threaded builds, and that both CFFI and setuptools are fairly aggressive about trying to use those.

In any case, we've been playing with it, and will continue to do so, but we absolutely will not be publishing CFFI wheels for the t ABI until the full test suite can pass. Possibly not even then, depending if forcing the GIL on in a FT build is actually acceptable to anyone- it likely wouldn't be to me as a user of a free-threaded Python. There's a ton of quite old init and dispatch code that will need to be very carefully evaluated and stress-tested to function reliably in a free-threaded environment.

IMO unstable t-ABI wheels are just an attractive nuisance, providing a false sense of security to folks playing around that their code might actually be safe, and leading to a lot of bug report noise at this stage, since the typical failure mode is almost always either a hang or a segfault.

I'm going to pin this issue to (hopefully) head off duplicates.

PS, since I couldn't resist playing with it again: as of 3.13rc1 with a default-GIL-disabled Python build against the current main branch, things have improved dramatically. With a few small hacks to forcefully disable all limited API support in CFFI, the vast majority of the CFFI test suite passes on x86_64 both with the implicit GIL disable in place, and with it force-disabled. A couple of currently-invalid assertions around limited API support, and a handful of segfaults in, surprise, surprise, the threading tests. It's great that it's a lot closer to working than it was a couple months ago, but real-world stuff still immediately segfaults.

nitzmahone commented 2 months ago

(also see #126 for overall status on that stuff)