JuliaIO / JLD2.jl

HDF5-compatible file format in pure Julia
Other
537 stars 84 forks source link

Don't map back to Julia composite type #504

Closed koehlerson closed 8 months ago

koehlerson commented 8 months ago

Let's say I have a struct called Foo and I save one instance to disk

julia> struct Foo
           x::Int
           z::Float64
       end

julia> jldopen("blah.jld2","w") do jld
           jld["test"] = Foo(1,2.0)
       end
Foo(1, 2.0)

if I close the session, open a new one and slightly redefine the type by only adding a parametrization (even though the data needed to reconstruct is still there) I cannot read the data in the form of x and z values

julia> using JLD2

julia> struct Foo{T1,T2}
           x::T1
           z::T2
       end

julia> file = jldopen("blah.jld2","r")
JLDFile /home/mkoehler/blah.jld2 (read-only)
 └─🔢 test

julia> file["test"]
┌ Warning: read type Foo is not a leaf type in workspace; reconstructing
└ @ JLD2 ~/.julia/packages/JLD2/u57Vt/src/data/reconstructing_datatypes.jl:306
Reconstruct@Foo((1, 2.0))

Is there a way such that I don't remap the stored data to Foo but instead just read the plain data, i.e. x and z in this case? Basically, what I want is HDF5.jl behavior as a possible option:

julia> h5open("blah.h5","w") do hdf5
           hdf5["test"] = [Foo(1,2.0)]
       end
1-element Vector{Foo{Int64, Float64}}:
 Foo{Int64, Float64}(1, 2.0)

julia> h5open("blah.h5","r") do hdf5
           hdf5["test"][]
       end
1-element Vector{NamedTuple{(:x, :z), Tuple{Int64, Float64}}}:
 (x = 1, z = 2.0)

If I open the file with HDF5 it works as I intend:

julia> h5open("blah.jld2","r") do jld2
           jld2["test"][]
       end
(x = 1, z = 2.0)

but I think it would be nice to have the optional type remapping when reading a file with JLD2. I know that an explicit type remapping is possible as well as doing a custom serialization, but as far as I know the custom serialization option always remaps to a composite type back and I have some use cases where I don't want this all the time, but instead I'm okay to look at (x=1,z=2.0).

JonasIsensee commented 8 months ago

Hi @koehlerson,

I will admit, that this is not obvious but with https://juliaio.github.io/JLD2.jl/dev/advanced/

you can do

julia> load("blah.jld2"; typemap=Dict("Main.Foo"=>JLD2.Upgrade(NamedTuple)))
Dict{String, Any} with 1 entry:
  "test" => (x = 1, z = 2.0)
koehlerson commented 8 months ago

Thanks for the hint!