search

nalepae / pandarallel

A simple and efficient tool to parallelize Pandas operations on all available CPUs
https://nalepae.github.io/pandarallel
BSD 3-Clause "New" or "Revised" License
3.59k stars 208 forks source link

Python 3.13 GIL-free #247

Open nalepae opened 11 months ago

nalepae commented 11 months ago

Python 3.13 may be released with an option to disable the GIL: https://discuss.python.org/t/a-steering-council-notice-about-pep-703-making-the-global-interpreter-lock-optional-in-cpython/30474

Currently, the Python GIL forces Pandarallel to split workers over processes, with slow and expensive inter-process communication.

A version of Python without the GIL would allow Pandarallel to split workers in multi-thread mode. It would avoid this inter-process and data duplication, and thus it may introduce a potentially massive performance bump!

Let's keep an eye on it!

(@till-m)

nalepae commented 5 months ago

Pandaral·lel is looking for a maintainer! If you are interested, please open an GitHub issue.