apache / arrow-julia

Official Julia implementation of Apache Arrow
https://arrow.apache.org/julia/
Other
284 stars 59 forks source link

colmetadata does not read custom metadata with multiple writes #485

Open JakeZw opened 1 year ago

JakeZw commented 1 year ago

This problem is described in more detail in discourse.

Specifically the following MWE works when the for loop executes once, but the colmetadata gives nothing when the for loop executes more than once.

`using Arrow using Tables

go to runtest.jl in Arrow package for hints on how to use metadata and colmetadata

cd(@DIR) filename = "test10.arrow" meta = ["file" => "test.arrow", "when" => "now", "why" => "gain experience"] metacol1 = ["id" => "chan1", "comment" => "Chan 1 comment"] metacol2 = ["id" => "chan2", "comment" => "Chan 2 comment"] writer = open(Arrow.Writer, filename, metadata=meta, colmetadata = Dict(:Column1 => metacol1, :Column2 => metacol2))

for i in 1:1 result = rand(2, 2) .+ i Arrow.write(writer, Tables.table(result)) #, header=[:a,:b]) end close(writer)

tbl = Arrow.Table(filename)

tblmeta = Arrow.getmetadata(tbl) tblmeta tblmeta["why"] tblmeta["when"]

tblmetachan1 = Arrow.getmetadata(tbl.Column1) tblmetachan1 tblmetachan1 == nothing `

For once through the for loop we get julia> tblmetachan1 = Arrow.getmetadata(tbl.Column1) Base.ImmutableDict{String, String} with 2 entries: "comment" => "Chan 1 comment" "id" => "chan1"

When we go through the for loop twice (or more) we get `julia> tblmetachan1 = Arrow.getmetadata(tbl.Column1)

julia> tblmetachan1

julia> tblmetachan1 == nothing true`