Multi-threaded multi-path Pathfinder broken with recent Transducers versions

[sethaxen]

Transducers v0.4.76 has caused our tests to fail (see first failure here).

Some not-necessarily-relevant notes:

• In addition to the merged PRs in those release notes, https://github.com/JuliaFolds/Transducers.jl/pull/553 was also merged, which supposedly resolved https://github.com/JuliaFolds/Transducers.jl/issues/552
• We currently use a deprecated method (from https://github.com/mlcolab/Pathfinder.jl/actions/runs/4920822614/jobs/8790013138#step:6:775):

  ┌ Warning: `reduce(rf, itr::Foldable; kw...)` is deprecated, use `foldxt(rf, itr; kw...)` instead.
  │   caller = multipathfinder(optim_fun::OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{LogDensityFunctionWithGradHess{var"#logp#44"{FullNormal}, var"#∇f#40"{var"#logp#44"{FullNormal}}, var"#Hf#41"{var"#logp#44"{FullNormal}}}}, Pathfinder.var"#grad#6"{LogDensityFunctionWithGradHess{var"#logp#44"{FullNormal}, var"#∇f#40"{var"#logp#44"{FullNormal}}, var"#Hf#41"{var"#logp#44"{FullNormal}}}}, Pathfinder.var"#hess#7"{LogDensityFunctionWithGradHess{var"#logp#44"{FullNormal}, var"#∇f#40"{var"#logp#44"{FullNormal}}, var"#Hf#41"{var"#logp#44"{FullNormal}}}}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, ndraws::Int64; init::Nothing, input::LogDensityFunctionWithGradHess{var"#logp#44"{FullNormal}, var"#∇f#40"{var"#logp#44"{FullNormal}}, var"#Hf#41"{var"#logp#44"{FullNormal}}}, nruns::Int64, ndraws_elbo::Int64, ndraws_per_run::Int64, rng::MersenneTwister, history_length::Int64, optimizer::LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, executor::SequentialEx{NamedTuple{(), Tuple{}}}, executor_per_run::SequentialEx{NamedTuple{(), Tuple{}}}, importance::Bool, kwargs::Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}) at multipath.jl:182
  └ @ Pathfinder ~/work/Pathfinder.jl/Pathfinder.jl/src/multipath.jl:182
  rng = TaskLocalRNG(): Error During Test at /home/runner/work/Pathfinder.jl/Pathfinder.jl/test/multipath.jl:31

[sethaxen]

Here's an MWE for this error:

using LogDensityProblems, Pathfinder, Transducers

struct StdNormalProblem
    n::Int
end
LogDensityProblems.capabilities(::Type{StdNormalProblem}) = LogDensityProblems.LogDensityOrder{1}()
LogDensityProblems.dimension(prob::StdNormalProblem) = prob.n
LogDensityProblems.logdensity(prob::StdNormalProblem, x) = -sum(abs2, x) / 2
LogDensityProblems.logdensity_and_gradient(prob::StdNormalProblem, x) = (-sum(abs2, x) / 2, -x)

ndraws = 100
nruns = 4
prob = StdNormalProblem(2)
multipathfinder(prob, ndraws; nruns)  # fine
multipathfinder(prob, ndraws; nruns, executor=ThreadedEx())  # errors, see below

error

```julia ERROR: MethodError: no method matching iterate(::Transducers.ProgressLoggingFoldable{Vector{Nothing}}) Closest candidates are: iterate(::Union{LinRange, StepRangeLen}) @ Base range.jl:880 iterate(::Union{LinRange, StepRangeLen}, ::Integer) @ Base range.jl:880 iterate(::T) where T<:Union{Base.KeySet{<:Any, <:Dict}, Base.ValueIterator{<:Dict}} @ Base dict.jl:698 ... Stacktrace: [1] iterate(::Base.Iterators.PartitionIterator{Transducers.ProgressLoggingFoldable{Vector{Nothing}}}) @ Base.Iterators ./iterators.jl:1319 [2] copyto!(dest::Vector{Vector{Nothing}}, src::Base.Iterators.PartitionIterator{Transducers.ProgressLoggingFoldable{Vector{Nothing}}}) @ Base ./abstractarray.jl:946 [3] _collect(cont::UnitRange{Int64}, itr::Base.Iterators.PartitionIterator{Transducers.ProgressLoggingFoldable{Vector{Nothing}}}, #unused#::Base.HasEltype, isz::Base.HasLength) @ Base ./array.jl:713 [4] collect(itr::Base.Iterators.PartitionIterator{Transducers.ProgressLoggingFoldable{Vector{Nothing}}}) @ Base ./array.jl:707 [5] split_into_chunks(coll::Transducers.ProgressLoggingFoldable{Vector{Nothing}}, sz::Int64) @ Transducers ~/.julia/packages/Transducers/IWhZW/src/reduce.jl:364 [6] _tcopy(xf::Map{Pathfinder.var"#32#37"{Int64, Int64, Random._GLOBAL_RNG, Int64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, SequentialEx{NamedTuple{(), Tuple{}}}, Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}, SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, ::Type{Vector}, reducible::Transducers.ProgressLoggingFoldable{Vector{Nothing}}, ::Transducers.SizeStable, ::Base.HasShape{1}; basesize::Int64, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Transducers ~/.julia/packages/Transducers/IWhZW/src/reduce.jl:358 [7] _tcopy(xf::Map{Pathfinder.var"#32#37"{Int64, Int64, Random._GLOBAL_RNG, Int64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, SequentialEx{NamedTuple{(), Tuple{}}}, Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}, SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, ::Type{Vector}, reducible::Transducers.ProgressLoggingFoldable{Vector{Nothing}}, ::Transducers.SizeStable, ::Base.HasShape{1}) @ Transducers ~/.julia/packages/Transducers/IWhZW/src/reduce.jl:356 [8] tcopy(xf::Map{Pathfinder.var"#32#37"{Int64, Int64, Random._GLOBAL_RNG, Int64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, SequentialEx{NamedTuple{(), Tuple{}}}, Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}, SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, T::Type, reducible::Transducers.ProgressLoggingFoldable{Vector{Nothing}}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Transducers ~/.julia/packages/Transducers/IWhZW/src/reduce.jl:354 [9] tcopy(xf::Map{Pathfinder.var"#32#37"{Int64, Int64, Random._GLOBAL_RNG, Int64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, SequentialEx{NamedTuple{(), Tuple{}}}, Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}, SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, T::Type, reducible::Transducers.ProgressLoggingFoldable{Vector{Nothing}}) @ Transducers ~/.julia/packages/Transducers/IWhZW/src/reduce.jl:354 [10] tcollect(xf::Map{Pathfinder.var"#32#37"{Int64, Int64, Random._GLOBAL_RNG, Int64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, SequentialEx{NamedTuple{(), Tuple{}}}, Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}, SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, reducible::Transducers.ProgressLoggingFoldable{Vector{Nothing}}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Transducers ~/.julia/packages/Transducers/IWhZW/src/reduce.jl:434 [11] tcollect(xf::Map{Pathfinder.var"#32#37"{Int64, Int64, Random._GLOBAL_RNG, Int64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, SequentialEx{NamedTuple{(), Tuple{}}}, Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}, SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, reducible::Transducers.ProgressLoggingFoldable{Vector{Nothing}}) @ Transducers ~/.julia/packages/Transducers/IWhZW/src/reduce.jl:434 [12] collect(itr::Transducers.Eduction{Transducers.Reduction{Map{Pathfinder.var"#32#37"{Int64, Int64, Random._GLOBAL_RNG, Int64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, SequentialEx{NamedTuple{(), Tuple{}}}, Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}, SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, Transducers.BottomRF{Completing{typeof(push!!)}}}, Transducers.ProgressLoggingFoldable{Vector{Nothing}}}, ex::ThreadedEx{NamedTuple{(), Tuple{}}}) @ Folds.Implementations ~/.julia/packages/Folds/ZayPF/src/collect.jl:12 [13] collect(itr::Transducers.Eduction{Transducers.Reduction{Map{Pathfinder.var"#32#37"{Int64, Int64, Random._GLOBAL_RNG, Int64, Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, SequentialEx{NamedTuple{(), Tuple{}}}, Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}, SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}, Transducers.BottomRF{Completing{typeof(push!!)}}}, Transducers.ProgressLoggingFoldable{Vector{Nothing}}}, ex::PreferParallel{NamedTuple{(), Tuple{}}}) @ Folds.Implementations ~/.julia/packages/Folds/ZayPF/src/collect.jl:9 [14] multipathfinder(optim_fun::SciMLBase.OptimizationFunction{true, SciMLBase.NoAD, Pathfinder.var"#f#5"{StdNormalProblem}, Pathfinder.var"#grad#6"{StdNormalProblem}, Pathfinder.var"#hess#7"{StdNormalProblem}, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, typeof(SciMLBase.DEFAULT_OBSERVED_NO_TIME), Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, ndraws::Int64; init::Nothing, input::StdNormalProblem, nruns::Int64, ndraws_elbo::Int64, ndraws_per_run::Int64, rng::Random._GLOBAL_RNG, history_length::Int64, optimizer::Optim.LBFGS{Nothing, LineSearches.InitialStatic{Float64}, LineSearches.MoreThuente{Float64}, Optim.var"#20#22"}, executor::PreferParallel{NamedTuple{(), Tuple{}}}, executor_per_run::SequentialEx{NamedTuple{(), Tuple{}}}, importance::Bool, kwargs::Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:dim,), Tuple{Int64}}}) @ Pathfinder ~/.julia/packages/Pathfinder/BUjTN/src/multipath.jl:179 [15] multipathfinder(ℓ::StdNormalProblem, ndraws::Int64; input::StdNormalProblem, kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:nruns, :executor), Tuple{Int64, PreferParallel{NamedTuple{(), Tuple{}}}}}}) @ Pathfinder ~/.julia/packages/Pathfinder/BUjTN/src/multipath.jl:130 [16] top-level scope @ REPL[13]:1 ```

@marcobonici is this the error you observed when using Pathfinder with executor=PreferParallel()?

[sethaxen]

This new failure in Transducers might be related: https://github.com/JuliaFolds/Transducers.jl/issues/557

[marcobonici]

Hi @sethaxen , yes, this is the same error I found. In order to deal with it I just downgraded to the previous version of Transducers.jl, but I can understand that from a package mantainer point of view this is not satisfactory :sweat_smile:

[sethaxen]

The new issue to follow is https://github.com/JuliaFolds2/Transducers.jl/issues/10