JuliaFolds / Transducers.jl

Efficient transducers for Julia
https://juliafolds.github.io/Transducers.jl/dev/
MIT License
433 stars 24 forks source link

`foldxd` fails over `Iterators.Product` #552

Closed sethaxen closed 1 year ago

sethaxen commented 1 year ago

I first came across this trying to use FLoops to parallelize execution over an Iterators.Product. It worked fine for sequential or multithreaded execution but not for distributed execution:

julia> using Transducers

julia> trans = Iterators.product(1:10, 1:10) |> Transducers.Map(sum);

julia> foldl(+, trans)
1100

julia> foldxt(+, trans)
1100

julia> foldxd(+, trans)
ERROR: MethodError: no method matching firstindex(::Base.Iterators.ProductIterator{Tuple{UnitRange{Int64}, UnitRange{Int64}}})

Closest candidates are:
  firstindex(::Any, ::Any)
   @ Base abstractarray.jl:443
  firstindex(::Union{Tables.AbstractColumns, Tables.AbstractRow})
   @ Tables ~/.julia/packages/Tables/T7rHm/src/Tables.jl:182
  firstindex(::AbstractChar)
   @ Base char.jl:200
  ...

Stacktrace:
 [1] dtransduce(xform::Map{typeof(sum)}, step::Function, init::Transducers.InitOf{Transducers.DefaultInitOf}, coll0::Base.Iterators.ProductIterator{Tuple{UnitRange{Int64}, UnitRange{Int64}}}; simd::Val{false}, basesize::Nothing, threads_basesize::Nothing, pool::Distributed.WorkerPool, _remote_reduce::Function)
   @ Transducers ~/.julia/packages/Transducers/iubDU/src/dreduce.jl:78
 [2] dtransduce(xform::Map{typeof(sum)}, step::Function, init::Transducers.InitOf{Transducers.DefaultInitOf}, coll0::Base.Iterators.ProductIterator{Tuple{UnitRange{Int64}, UnitRange{Int64}}})
   @ Transducers ~/.julia/packages/Transducers/iubDU/src/dreduce.jl:50
 [3] foldxd(step::Function, xform::Map{typeof(sum)}, itr::Base.Iterators.ProductIterator{Tuple{UnitRange{Int64}, UnitRange{Int64}}}; init::Transducers.InitOf{Transducers.DefaultInitOf}, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Transducers ~/.julia/packages/Transducers/iubDU/src/dreduce.jl:40
 [4] foldxd(step::Function, xform::Map{typeof(sum)}, itr::Base.Iterators.ProductIterator{Tuple{UnitRange{Int64}, UnitRange{Int64}}})
   @ Transducers ~/.julia/packages/Transducers/iubDU/src/dreduce.jl:40
 [5] foldxd(step::typeof(+), foldable::Transducers.Eduction{Transducers.Reduction{Map{typeof(sum)}, Transducers.BottomRF{Completing{typeof(push!!)}}}, Base.Iterators.ProductIterator{Tuple{UnitRange{Int64}, UnitRange{Int64}}}}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Transducers ~/.julia/packages/Transducers/iubDU/src/dreduce.jl:42
 [6] foldxd(step::typeof(+), foldable::Transducers.Eduction{Transducers.Reduction{Map{typeof(sum)}, Transducers.BottomRF{Completing{typeof(push!!)}}}, Base.Iterators.ProductIterator{Tuple{UnitRange{Int64}, UnitRange{Int64}}}})
   @ Transducers ~/.julia/packages/Transducers/iubDU/src/dreduce.jl:42
 [7] top-level scope
   @ REPL[36]:1
MasonProtter commented 1 year ago

Should be fixed in https://github.com/JuliaFolds/Transducers.jl/pull/553/commits/2fe7849c591ac5e56d40aae25465bf76550d608b, will be available once https://github.com/JuliaFolds/Transducers.jl/pull/553 merges, though I'm not super happy with the implementation since it uses Iterators.partition instead of halve.