search

xiaodaigh / JDF.jl

Julia DataFrames serialization format
MIT License
90 stars 9 forks source link

error when trying to save df that has array of substrings as column #47

Closed ym-han closed 4 years ago

ym-han commented 4 years ago

Hi,

Nice work on this package. Just wanted to report that when I tried saving a df that had Array{SubString{String},1} for one of its columns, I got the following error:

ERROR: TaskFailedException: ArgumentError: buffer eltype must beisbits` type Stacktrace: [1] compress!(::Array{UInt8,1}, ::Ptr{SubString{String}}, ::Int64; level::Int64, shuffle::Bool, itemsize::Int64) at /users/yh31/.julia/packages/Blosc/Six7M/src/Blosc.jl:60 [2] compress!(::Array{UInt8,1}, ::Ptr{SubString{String}}, ::Int64) at /users/yh31/.julia/packages/Blosc/Six7M/src/Blosc.jl:58 [3] compress(::Ptr{SubString{String}}, ::Int64; kws::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /users/yh31/.julia/packages/Blosc/Six7M/src/Blosc.jl:86 [4] compress(::Ptr{SubString{String}}, ::Int64) at /users/yh31/.julia/packages/Blosc/Six7M/src/Blosc.jl:85 [5] compress(::Array{SubString{String},1}; kws::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /users/yh31/.julia/packages/Blosc/Six7M/src/Blosc.jl:92 [6] compress(::Array{SubString{String},1}) at /users/yh31/.julia/packages/Blosc/Six7M/src/Blosc.jl:91 [7] compress_then_write(::Array{SubString{String},1}, ::BufferedStreams.BufferedOutputStream{IOStream}) at /users/yh31/.julia/packages/JDF/Na1Xj/src/compress_then_write.jl:8 [8] macro expansion at /users/yh31/.julia/packages/JDF/Na1Xj/src/savejdf.jl:69 [inlined] [9] (::JDF.var"#49#52"{String,DataFrame,String,Int64})() at ./threadingconstructs.jl:169 Stacktrace: [1] wait at ./task.jl:267 [inlined] [2] fetch(::Task) at ./task.jl:282 [3] _broadcast_getindex_evalf at ./broadcast.jl:648 [inlined] [4] _broadcast_getindex at ./broadcast.jl:621 [inlined] [5] getindex at ./broadcast.jl:575 [inlined] [6] copyto_nonleaf!(::Array{NamedTuple{(:string_compressed_bytes, :string_len_bytes, :rle_bytes, :rle_len, :type, :len),Tuple{Int64,Int64,Int64,Int64,DataType,Int64}},1}, ::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1},Tuple{Base.OneTo{Int64}},typeof(fetch),Tuple{Base.Broadcast.Extruded{Array{Any,1},Tuple{Bool},Tuple{Int64}}}}, ::Base.OneTo{Int64}, ::Int64, ::Int64) at ./broadcast.jl:1026 [7] copy at ./broadcast.jl:880 [inlined] [8] materialize at ./broadcast.jl:837 [inlined] [9] savejdf(::String, ::DataFrame; verbose::Bool) at /users/yh31/.julia/packages/JDF/Na1Xj/src/savejdf.jl:76 [10] savejdf at /users/yh31/.julia/packages/JDF/Na1Xj/src/savejdf.jl:48 [inlined] [11] savejdf(::DataFrame, ::String) at /users/yh31/.julia/packages/JDF/Na1Xj/src/savejdf.jl:45

`

This error went away when I converted the array to an array of strings before saving the df.

xiaodaigh commented 4 years ago

I see. Let me test this.

xiaodaigh commented 4 years ago

Version 0.2.19 (registering now) has supported for SubString, but it will be loaded back as String. Thanks for the feature request.

ym-han commented 4 years ago

Wow, thanks for being so responsive and adding this functionality!

On Fri, 14 Aug 2020 at 23:13, evalparse notifications@github.com wrote:

Version 0.2.19 (registering now) has supported for SubString, but it will be loaded back as String. Thanks for the feature request.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/xiaodaigh/JDF.jl/issues/47#issuecomment-674340264, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFCM7HGCAGCE4SJVWR3LUK3SAX4M3ANCNFSM4P7CP2MQ .

xiaodaigh commented 4 years ago

Does it work for you?