JuliaFolds2 / OhMyThreads.jl

Simple multithreading in julia
https://juliafolds2.github.io/OhMyThreads.jl/
MIT License
137 stars 9 forks source link

Allow setting `OutputElementType` within `@tasks` #92

Open carstenbauer opened 6 months ago

carstenbauer commented 6 months ago

See #TODO in macro_impl.jl. Especially relevant because it's sometimes required:

julia> @tasks for i in 1:10
           @set scheduler=:greedy
           @set collect=true
           sin(i)
       end
ERROR: Greedy scheduler isn't supported with `tmap` unless you provide an `OutputElementType` argument, since the greedy schedule requires a commutative reducing operator.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] tmap(::Function, ::UnitRange{Int64}; scheduler::Symbol, kwargs::@Kwargs{})
   @ OhMyThreads.Implementation ~/repos/OhMyThreads.jl/src/implementation.jl:339
 [3] macro expansion
   @ ~/repos/OhMyThreads.jl/src/macro_impl.jl:50 [inlined]
 [4] top-level scope
   @ REPL[26]:1
MasonProtter commented 6 months ago

We should probably just generally make @set expr pass expr as a keyword argument to tmapreduce. That way you could also splat keyword arguments like

f(v; kw...) = @ tasks for x in v
    @set kw...
    @set collect=true
    sin(x)
end

That does raise the question of @set collect though. Should that be a kwarg of tmapreduce? Feels kinda weird and wrong to include it as a kwarg of that, but I guess we could have a separate tasks_mapreduce function that has collect as a kwarg.

carstenbauer commented 18 hours ago

Update: We decided to pass @set expr as keyword arguments to the scheduler rather than tmapreduce. Hence, we now get

julia> @tasks for i in 1:10
           @set scheduler=:greedy
           @set collect=true
           @set outputtype=Float64
           sin(i)
       end
ERROR: MethodError: no method matching GreedyScheduler(; outputtype::DataType)

Closest candidates are:
  GreedyScheduler(; ntasks, nchunks, chunksize, chunking, split) got unsupported keyword argument "outputtype"
   @ OhMyThreads ~/.julia/packages/OhMyThreads/eiaNP/src/schedulers.jl:298
  GreedyScheduler(::Int64, ::Integer, ::Integer, ::Union{ChunkSplitters.Split, Symbol}; chunking) got unsupported keyword argument "outputtype"
   @ OhMyThreads ~/.julia/packages/OhMyThreads/eiaNP/src/schedulers.jl:271

Stacktrace:
 [1] kwerr(kw::@NamedTuple{outputtype::DataType}, args::Type)
   @ Base ./error.jl:165
 [2] scheduler_from_symbol(::Type{Val{:greedy}}; kwargs::@Kwargs{outputtype::DataType})
   @ OhMyThreads.Schedulers ~/.julia/packages/OhMyThreads/eiaNP/src/schedulers.jl:355
 [3] scheduler_from_symbol(s::Symbol; kwargs::@Kwargs{outputtype::DataType})
   @ OhMyThreads.Schedulers ~/.julia/packages/OhMyThreads/eiaNP/src/schedulers.jl:352
 [4] scheduler_from_symbol
   @ ~/.julia/packages/OhMyThreads/eiaNP/src/schedulers.jl:352 [inlined]
 [5] #_scheduler_from_userinput#20
   @ ~/.julia/packages/OhMyThreads/eiaNP/src/implementation.jl:44 [inlined]
 [6] _scheduler_from_userinput
   @ ~/.julia/packages/OhMyThreads/eiaNP/src/implementation.jl:39 [inlined]
 [7] tmap(::Function, ::UnitRange{Int64}; scheduler::Symbol, kwargs::@Kwargs{outputtype::DataType})
   @ OhMyThreads.Implementation ~/.julia/packages/OhMyThreads/eiaNP/src/implementation.jl:342
 [8] macro expansion
   @ ~/.julia/packages/OhMyThreads/eiaNP/src/macro_impl.jl:98 [inlined]
 [9] top-level scope
   @ REPL[4]:1

We could add explicit support for outputtype (like for collect or reduce) or we could pass the keyword arguments not just to the scheduler but also to tmapreduce and let them figure out which keyword arguments are relevant to them (perhaps with an extra flag, because we generally want to throw errors for unsupported keyword arguments).