Open indigoviolet opened 1 year ago
Multiprocessing would likely content with polars.
Besides that it must clone data and has a terrible start up/break down cost
We can allow multithreading and use polars thread pool. This will have benefit if your python function releases the GIL.
This relates a bit to my issue here: https://github.com/pola-rs/polars/issues/6157#issuecomment-1377420903 Just the inverse way of running Polars in multiprocessing
Problem description
I wish I could tell Polars that my
apply
function or mymap
function is safe to run in parallel, and it would automatically usemultiprocessing
to run it over my column. This seems like a common case which could be made very easy to use.