xiaodaigh / JDF.jl

Julia DataFrames serialization format
MIT License
88 stars 9 forks source link

Version Stable Metadata Format #43

Closed ro-ble closed 4 years ago

ro-ble commented 4 years ago

Hi Xiaodaigh,

Currently, the metadata of a JDF file is stored using Serialization.jl. Reading the metadata is fast and performat. However it is not guaranteed to be readable across Julia versions. From my understanding, the metadata file is crucial for reading the JDF file. Therefore, it would make sense to use a version stable serialization for the metadata file.

The main benefit of Serialization.jl is, that it already ships with Julia. There is no need to use an additional Package for reading the metadata. Using JSON/BSON for that purpose would probably decrease speed and increase compile times and dependencies. As an alternative to that, I made some experiments:

julia> metadata = (field1 = v"1.1", field2 = "HelloWorld", field3 = 42)
(field1 = v"1.1.0", field2 = "HelloWorld", field3 = 42)

julia> const metadata_t = typeof(metadata)
NamedTuple{(:field1, :field2, :field3),Tuple{VersionNumber,String,Int64}}

julia> open("C:\\Users\\me\\Desktop\\metadata.jl", "w") do io
           write(io, repr(metadata))
       end;

julia> metadata2 = include("C:\\Users\\me\\Desktop\\metadata.jl")
(field1 = v"1.1.0", field2 = "HelloWorld", field3 = 42)

julia> isa(metadata2, metadata_t)
true

What do you think about that? It uses NamedTouple syntax and would most probably be version stable. I am curious about how this would perform compared to JSON/BSON.

ro-ble commented 4 years ago

Duplicate of #25

xiaodaigh commented 4 years ago

repr(metadata)

I learned something new. I wanted to make JDF cross-language as well, so I will need to assess. Thanks

xiaodaigh commented 4 years ago

I will close it for now as repr doesn't support cross-language as well BSON