JuliaFolds2 / OhMyThreads.jl

Simple multithreading in julia
https://juliafolds2.github.io/OhMyThreads.jl/
MIT License
139 stars 9 forks source link

tcollect fails with nested generators #113

Open juliohm opened 3 months ago

juliohm commented 3 months ago

MWE:

using Transducers
using OhMyThreads

f(xs) = Transducers.tcollect(abs(x) for x in xs)
g(xs) = OhMyThreads.tcollect(abs(x) for x in xs)

f((x for x in rand(100))) # works as expected
g((x for x in rand(100))) # fails with the error below
ERROR: MethodError: no method matching tmap(::typeof(identity), ::Base.Generator{Base.Generator{Vector{Float64}, typeof(identity)}, var"#13#14"})

Closest candidates are:
  tmap(::Any, ::Union{ChunkSplitters.Chunk, AbstractArray}, AbstractArray...; scheduler, kwargs...)
   @ OhMyThreads ~/.julia/packages/OhMyThreads/PtzLw/src/implementation.jl:323
  tmap(::Any, ::Type{T}, ::AbstractArray, ::AbstractArray...; kwargs...) where T
   @ OhMyThreads ~/.julia/packages/OhMyThreads/PtzLw/src/implementation.jl:318

Stacktrace:
 [1] tcollect(A::Base.Generator{Base.Generator{Vector{Float64}, typeof(identity)}, var"#13#14"}; kwargs::@Kwargs{})
   @ OhMyThreads.Implementation ~/.julia/packages/OhMyThreads/PtzLw/src/implementation.jl:472
 [2] g(xs::Base.Generator{Vector{Float64}, typeof(identity)})
   @ Main ./REPL[23]:1
 [3] top-level scope
   @ REPL[27]:1
carstenbauer commented 2 months ago

The issue here is the nesting of generators.

OhMyThreads.tcollect(abs(x) for x in rand(100))

works just fine.

carstenbauer commented 2 months ago

I'm not sure we want to or even can support this properly.