search

JuliaML / MLUtils.jl

Utilities and abstractions for Machine Learning tasks
MIT License
107 stars 20 forks source link

"failed to start primary task" with Julia 1.9 and nthreads(:interactive) > 0 #158

Closed MarkusSchildhauer closed 1 year ago

MarkusSchildhauer commented 1 year ago

Reproducible on Julia 1.9 with:

julia --threads 2,1 -e "using MLUtils"  
┌ Error: failed to start primary task
└ @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/root_spawners.jl:34
ERROR: InitError: TaskFailedException

    nested task error: BoundsError: attempt to access 2-element Vector{Union{Nothing, Channel{Any}}} at index [3]
    Stacktrace:
     [1] setindex!
       @ ./array.jl:969 [inlined]
     [2] (::FoldsThreads.Implementations.var"#29#31")()
       @ FoldsThreads.Implementations ~/.julia/packages/FoldsThreads/aWnRU/src/root_spawners.jl:62
     [3] macro expansion
       @ ~/.julia/packages/FoldsThreads/aWnRU/src/utils.jl:6 [inlined]
...

I reported it already at FoldsThreads then at Flux. How can we tackle it?

ToucheSir commented 1 year ago

Ordinarily I would close the Flux issue as a duplicate of this one since the issue there comes from MLUtils depending on FoldsThreads, but since that one has more info I'll be closing this instead.

MarkusSchildhauer commented 1 year ago

@ToucheSir, sorry about this. But I did provide the root cause and an idea how to fix in the right place (https://github.com/JuliaFolds/FoldsThreads.jl/issues/21) already 2 weeks ago. But since it seems to be dead repo, I thought working on Fix/PR does not make sense. I might be necessary to solve it via MLUtils dependencies.

CarloLucibello commented 1 year ago

given the unmaintained state of FoldsThreads.jl and since it seems it has not been forked in https://github.com/JuliaFolds2, we should consider switching to some other backend for the parallelization of the DataLoader.

I remember @lorenzoh compared a few options, but I cannot find the issue/PR

ToucheSir commented 1 year ago

But since it seems to be dead repo, I thought working on Fix/PR does not make sense.

The JuliaFolds ecosystem is under new ownership now, but they may not be subscribed to issue notifications on FoldsThreads. I've pinged them elsewhere, but that shouldn't preclude us from exploring other options.

MarkusSchildhauer commented 1 year ago

Fixed in https://github.com/JuliaFolds2/FoldsThreads.jl/pull/2