List existing packages that would benefit from nogil

[gabrieldemarmiesse]

Hi @colesbury , thank you for your work!

I would like very much to see this project succeed. One common counter-argument to merge this work into CPython is "this will only benefit a subset of Python users, only those who care about multithreading".

While it's true, free-threading could also be used in common libraries, thus impacting many users, with 3 categories of libs that would benefit:

• The library already use threads and nogil will likely make some features faster.
• The library doesn't use threads, but nogil would allow them to rewrite some features with multithreading, thus making it faster.
• The library uses multithreading with a low-level language/multiprocessing and using nogil would allow a rewrite and lower the maintenance burden.

Those libraries could be in pypi, or even among the standard library. Giving a list of packages that would be in one of those three categories, with the number of daily downloads, would likely make a great argument in favor in nogil.

This might not be easy to provide a list, notably because sometimes, we would need to ask the core devs directly, but it can be done.

If you think this is helpful, I have some experience with the python open source ecosystem, I could go around, ask relevant lib maintainers and establish the list mentionned above.

Best, Gabriel

[SmartAudioTools]

I think PyQt in Qt for python, could greatly benefit from python nogil . I've been developing applications in PyQt and the gil generates a lot of problems for using CPUs efficiently (one thread for the GUI, and others for I/O and CPU bound calculations). I can't bring myself to port everything to C++ and I'm putting a lot of hope into integrating nogil into cpython.

[gabrieldemarmiesse]

Thanks @SmartAudioTools , if @colesbury if ok to establish a list of packages that would benefit from nogil, I will definitly ask the PyQt guys :)

[colesbury]

Hi @gabrieldemarmiesse, thanks for reaching out! The PEP includes quotes from a few library authors/maintainers along those lines (NumPy, SciPy, scikit-learn, CellProfiler, PyTorch). A few more have commented on the discussion forum. I'm not sure if a longer how much a longer list of packages will be critical to the SC's decision, but it probably can't hurt. The comments lately have been focused on core dev maintenance and the impact on the Faster CPython effort.

Regarding Qt, Christian Tismer has a branch of PySide6 that works with nogil-3.9: https://github.com/ctismer/pyside6-dev/tree/dev-nogil-patches.

[gabrieldemarmiesse]

Thanks, indeed the PEP has quite a few examples. Maybe what's missing are maintainers of packages that are not related to scientific computing, for instance PyQt or pygame (I know pygame and the GIL aren't really friends).

I will establish a list of selected popular packages unrelated to scientific computing and try to get their feedback on this. I'll keep you updated in this issue.

Best of luck with the rest!

[SmartAudioTools]

Maybe you should also contact the guys at Qt for Python (PySide). In our case of use (https://www.smartaudiotools.com) , it's a real challenge to build with GIL a software with a lot of python computation on CPU and inputs/outputs that have to react quickly. Today, the most obvious solution is to recode everything in C++, but we'd rather not do that. We have high hopes for the integration of nogil into python 3.13.

[gabrieldemarmiesse]

Hi @colesbury

I've done a first round of asking around. I've put in a spreadsheet the info if you want to discuss with the maintainers directly. I tried to answer the questions with the best of my knowledge, but don't hesitate to chime in!

https://docs.google.com/spreadsheets/d/10bpwm8z5sSwBTUlGcCB2OM5oKRNkd4wV822QbcDY7z8/edit?usp=sharing

I'll ask more projects later on when I have a bit of time.

I believe it's quite interesting to discuss each use case individually with people who likely wouldn't have voiced their opinion in the main discussion in python.org.