search

JuliaIO / JLD.jl

Saving and loading julia variables while preserving native types
MIT License
277 stars 55 forks source link

Trouble rescuing old Nullable using readas and translate #180

Open pearlzli opened 7 years ago

pearlzli commented 7 years ago

The type definition for Nullable changed from Julia v0.5 to v0.6. I have some Nullables which I saved in a JLD file while running v0.5, and I'm now trying to read them in in v0.6. I'm following the instructions in the Rescuing Old Types section of the documentation, but it doesn't appear to be working or I may be doing something incorrectly.

I first save a Nullable{Int} in v0.5:

julia> using JLD

julia> save("test.jld", "x", Nullable{Int}(5))

Then in v0.6, I attempt to read it in but still get an error:

julia> using JLD

julia> struct OldNullable{T}
       isnull::Bool
       value::T
       end

julia> JLD.readas(x::OldNullable) = Nullable(x.value, !x.isnull)

julia> translate("Nullable", "OldNullable")
"OldNullable"

julia> load("test.jld", "x")
WARNING: Compat.ASCIIString is deprecated, use String instead.
  likely near no file:0
WARNING: Compat.UTF8String is deprecated, use String instead.
  likely near no file:0
WARNING: Compat.ASCIIString is deprecated, use String instead.
  likely near no file:0
in jldatatype at /home/rcepxl10/.julia/v0.6/JLD/src/jld_types.jl
WARNING: Compat.UTF8String is deprecated, use String instead.
  likely near no file:0
in jldatatype at /home/rcepxl10/.julia/v0.6/JLD/src/jld_types.jl
ERROR: stored type Base.Nullable{Core.Int64} does not match currently loaded type
Stacktrace:
 [1] jldatatype(::JLD.JldFile, ::HDF5.HDF5Datatype) at /home/rcepxl10/.julia/v0.6/JLD/src/jld_types.jl:737
 [2] read(::JLD.JldDataset) at /home/rcepxl10/.julia/v0.6/JLD/src/JLD.jl:368
 [3] read(::JLD.JldFile, ::String) at /home/rcepxl10/.julia/v0.6/JLD/src/JLD.jl:344
 [4] (::JLD.##41#42{String})(::JLD.JldFile) at /home/rcepxl10/.julia/v0.6/JLD/src/JLD.jl:1239
 [5] #jldopen#11(::Array{Any,1}, ::Function, ::JLD.##41#42{String}, ::String, ::Vararg{String,N} where N) at /home/rcepxl10/.julia/v0.6/JLD/src/JLD.jl:243
 [6] load(::FileIO.File{FileIO.DataFormat{:JLD}}, ::String) at /home/rcepxl10/.julia/v0.6/JLD/src/JLD.jl:1238
 [7] #load#13(::Array{Any,1}, ::Function, ::String, ::String, ::Vararg{String,N} where N) at /home/rcepxl10/.julia/v0.6/FileIO/src/loadsave.jl:52
 [8] load(::String, ::String) at /home/rcepxl10/.julia/v0.6/FileIO/src/loadsave.jl:52

In v0.6, I'm using JLD on master. Thanks in advance for your help!

pearlzli commented 7 years ago

The problem appears to be that you have to specify the full module path for each type in the call to translate. In v0.6, this works for me:

julia> using JLD

julia> struct OldNullable{T}
       isnull::Bool
       value::T
       end

julia> JLD.readas(x::OldNullable) = Nullable(x.value, !x.isnull)

julia> translate("Base.Nullable{Core.Int64}", "OldNullable{Core.Int64}")
"OldNullable{Core.Int64}"

julia> load("test.jld", "x")
Nullable{Int64}(5)

I think this point would be worth clarifying in the documentation.

Also, is there any way to allow this call to translate to be parametric, so you don't have separately call translate("Base.Nullable{Core.Float64}", "OldNullable{Core.Float64}"), translate("Base.Nullable{MyType}", "OldNullable{MyType}"), etc?

timholy commented 7 years ago

Nice detective work! Can you add that to the documentation? Just click on the documentation source file, click the little pencil icon, and then you can copy/paste.

It's been ages since I've looked at translate, but from the signature there doesn't seem to be a way. You could perhaps explore adding a regular-expression variant of translate?

adicunningham commented 6 years ago

Hi,

I'm having trouble loading old 0.5 JLD files with Nullable{Date} types and was redirected here, wondering if there is something I'm doing wrong.

My original post in Julia discourse outlines the problem. Rescuing old Julia 0.5 Nullable{Date} type in 0.6

Basically loading the Julia 0.5 type with a Nullable{Date} fails with the following error:


stored type foo5 does not match currently loaded type

Stacktrace:
 [1] jldatatype(::JLD.JldFile, ::HDF5.HDF5Datatype) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\jld_types.jl:721
 [2] read(::JLD.JldDataset) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:368
 [3] read(::JLD.JldFile, ::String) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:344
 [4] (::JLD.##41#42{String})(::JLD.JldFile) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:1241
 [5] #jldopen#11(::Array{Any,1}, ::Function, ::JLD.##41#42{String}, ::String, ::Vararg{String,N} where N) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:243
 [6] load(::FileIO.File{FileIO.DataFormat{:JLD}}, ::String) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\JLD\src\JLD.jl:1240
 [7] #load#13(::Array{Any,1}, ::Function, ::String, ::String, ::Vararg{String,N} where N) at C:\Program Files\ReSolver.DistributedJulia\packages\v0.6\FileIO\src\loadsave.jl:113
 [8] from_file(::String) at C:\Projects\Reinsurance\ReSolver\Feature\DevJulia6\Distributed\Endurance.Re.ReSolver.Distributed.Julia\src\core\04_serialize.jl:57```
pearlzli commented 6 years ago

@adicunningham I just posted a response to your Discourse thread (not sure where the best place to continue this discussion is?)