JuliaArrays / AxisArrays.jl

Performant arrays where each dimension can have a named axis with values
http://JuliaArrays.github.io/AxisArrays.jl/latest/
Other
200 stars 41 forks source link

Stackoverflow error when doing cat on array of AxisArrays #203

Open mgmverburg opened 3 years ago

mgmverburg commented 3 years ago

Using AxisArrays v0.4.4, I encountered a StackOverflow error when doing concatenation with a relatively large array as below:

stackoverflowarray = [AxisArray(rand(500,4)) for i in 1:1048]
cat(stackoverflowarray...; dims=3)

This is actually not an uncommon size in MCMC methods, since 500x4 is just for 4 chains with 500 samples each, and 1048 would be the number of parameters.

Additionally, while doing the cat operation, I will see my RAM consistently climb for about 6GB until the stackoverflow error is thrown. This is probably not the cause though, but it might be correlated/side-effect, because I still have more RAM available, since I have seen julia climb to 12GB at times, and now it is at a total of 7GB when it throws the error.

Lastly, note that if I just do this:

nostackoverflowarray = [AxisArray(rand(500,4)) for i in 1:1048]
cat(nostackoverflowarray...; dims=3)

There is no stackoverflow errors, it finishes in a second, and the operation consumes maybe 25 MB

StackoverflowError: ERROR: StackOverflowError: Stacktrace: [1] _zip_iterate_all(::NTuple{1048,Tuple{Int64,Int64}}, ::NTuple{1048,Tuple{}}) at ./iterators.jl:341 [2] iterate(::Base.Iterators.Zip{NTuple{1048,Tuple{Int64,Int64}}}) at ./iterators.jl:334 [3] sizes(::AxisArray{Float64,2,Array{Float64,2},Tuple{Axis{:row,Base.OneTo{Int64}},Axis{:col,Base.OneTo{Int64}}}}, ::Vararg{AxisArray{Float64,2,Array{Float64,2},Tuple{Axis{:row,Base.OneTo{Int64}},Axis{:col,Base.OneTo{Int64}}}},N} where N) at /home/michiel/.julia/packages/AxisArrays/FWWEV/src/combine.jl:12 [4] matchingdims(::NTuple{1048,AxisArray{Float64,2,Array{Float64,2},Tuple{Axis{:row,Base.OneTo{Int64}},Axis{:col,Base.OneTo{Int64}}}}}) at /home/michiel/.julia/packages/AxisArrays/FWWEV/src/combine.jl:13 [5] _cat(::Int64, ::AxisArray{Float64,2,Array{Float64,2},Tuple{Axis{:row,Base.OneTo{Int64}},Axis{:col,Base.OneTo{Int64}}}}, ::Vararg{AxisArray{Float64,2,Array{Float64,2},Tuple{Axis{:row,Base.OneTo{Int64}},Axis{:col,Base.OneTo{Int64}}}},N} where N) at /home/michiel/.julia/packages/AxisArrays/FWWEV/src/combine.jl:26 [6] cat(::AxisArray{Float64,2,Array{Float64,2},Tuple{Axis{:row,Base.OneTo{Int64}},Axis{:col,Base.OneTo{Int64}}}}, ::AxisArray{Float64,2,Array{Float64,2},Tuple{Axis{:row,Base.OneTo{Int64}},Axis{:col,Base.OneTo{Int64}}}}, ::Vararg{AxisArray{Float64,2,Array{Float64,2},Tuple{Axis{:row,Base.OneTo{Int64}},Axis{:col,Base.OneTo{Int64}}}},N} where N; dims::Int64) at /home/michiel/.julia/packages/AxisArrays/FWWEV/src/combine.jl:16 [7] top-level scope at REPL[4]:1