JuliaFolds2 / OhMyThreads.jl

Simple multithreading in julia
https://juliafolds2.github.io/OhMyThreads.jl/
MIT License
139 stars 9 forks source link

Support `minchunksize` #114

Open MasonProtter opened 2 months ago

MasonProtter commented 2 months ago

I think a nice additional option would be a minchunksize that goes with nchunks. Basically, the way I'd want it to work is

scheduler=StaticScheduler(nchunks=10, minchunksize=5)
tforeach(f, 1:3; scheduler)        # does not spawn tasks (serial fallback)
tforeach(f, 1:10; scheduler)       # spawns 2 tasks
tforeach(f, 1:20; scheduler)       # spawns 4 tasks
tforeach(f, 1:100; scheduler)      # spawns 10 tasks
tforeach(f, 1:10_000; scheduler)   # still spawns 10 tasks

spawn 10 tasks. I think this is really useful for the majority of naively parallel operations on well behaved data when the mapping / reducing functions are cheap, because you can basically say "I want this parallelized, but don't create more tasks than necessary"

carstenbauer commented 2 months ago

Sounds like a good idea to me.

carstenbauer commented 1 month ago

Might make sense to support this upstream in ChunkSplitters.jl (cc @lmiq).

carstenbauer commented 3 weeks ago

Should be easily doable now that we use ChunkSplitters v3.x, which has this as minsize.