tkf / ThreadsX.jl

Parallelized Base functions
MIT License
321 stars 10 forks source link

Iterators.partition gives suboptimal partitions for multithreading #168

Open nalimilan opened 3 years ago

nalimilan commented 3 years ago

I've noticed that foreach_array is partly inspired by ThreadTools' tforeach. I did the same for DataFrames.jl, and we noticed that Iterators.partition doesn't give very good results since the last chunk will generally be smaller than others, which is bad for load balancing. See https://github.com/JuliaData/DataFrames.jl/pull/2661#discussion_r593959480 for details and a possible solution. I hope this is useful.