xiaodaigh / JDF.jl

Julia DataFrames serialization format
MIT License
88 stars 9 forks source link

Error when writing DataFrame with column containing Vector{Int64} #83

Closed Para7etamol closed 1 year ago

Para7etamol commented 1 year ago

Hi there,

would love to use JDF.jl with my data :-) but:

JDF.save("df_with_array.jdf", DataFrame(a=[[1,2]]))

results in

nested task error: Type Array does not have a definite size.
Stacktrace:
 [1] sizeof(x::Type)
   @ Base ./essentials.jl:473
 [2] compress!(dest::Vector{UInt8}, src::Ptr{Vector{Int64}}, src_size::Int64)
   @ Blosc ~/.julia/packages/Blosc/jk4Np/src/Blosc.jl:74
 [3] compress(src::Ptr{Vector{Int64}}, src_size::Int64; kws::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Blosc ~/.julia/packages/Blosc/jk4Np/src/Blosc.jl:111
 [4] compress(src::Ptr{Vector{Int64}}, src_size::Int64)
   @ Blosc ~/.julia/packages/Blosc/jk4Np/src/Blosc.jl:109
 [5] compress(src::Vector{Vector{Int64}}; kws::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Blosc ~/.julia/packages/Blosc/jk4Np/src/Blosc.jl:117
 [6] compress(src::Vector{Vector{Int64}})
   @ Blosc ~/.julia/packages/Blosc/jk4Np/src/Blosc.jl:115
 [7] compress_then_write(b::Vector{Vector{Int64}}, io::BufferedStreams.BufferedOutputStream{IOStream})
   @ JDF ~/.julia/packages/JDF/SL7sz/src/compress_then_write.jl:7
 [8] macro expansion
   @ ~/.julia/packages/JDF/SL7sz/src/savejdf.jl:71 [inlined]
 [9] (::JDF.var"#47#50"{String, DataFrame, Symbol})()
   @ JDF ./threadingconstructs.jl:258

using

JDF v0.5.1

and

julia 1.8.5

Is there a workaround?

Greetings Para

xiaodaigh commented 1 year ago

The issue is that a=[[1,2]] is an array of array. An array is of variable size which issn't supported by jdf.jl but a=[1,2] is since Int is of fixed size.