search

JuliaIO / JLD.jl

Saving and loading julia variables while preserving native types
MIT License
278 stars 55 forks source link

JLD failure #249

Open montyvesselinov opened 5 years ago

montyvesselinov commented 5 years ago

A file generated (JLD.save) with the latest version of julia (1.1) and JLD (v0.9.1) cannot be loaded back using JLD.load. How is this possible?

ERROR: unrecognized HDF5 datatype class 4 Stacktrace: [1] error(::String, ::Int32) at ./error.jl:42 [2] jldatatype(::JLD.JldFile, ::HDF5.HDF5Datatype) at /Users/monty/.julia/packages/JLD/1BoSz/src/jld_types.jl:738 [3] read(::JLD.JldDataset) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:370 [4] read_ref(::JLD.JldFile, ::HDF5.HDF5ReferenceObj) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:502 [5] read_refs(::JLD.JldDataset, ::Type{Any}, ::Int32, ::Int32, ::Tuple{Int64}) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:475 [6] read_array(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Int32, ::Int32, ::Tuple{Int64}) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:409 [7] read_array(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Int32, ::Int32) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:408 [8] read(::JLD.JldDataset) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:372 [9] read_ref(::JLD.JldFile, ::HDF5.HDF5ReferenceObj) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:502 [10] jlconvert(::Type{JLD.AssociativeWrapper{Symbol,Any,Dict{Symbol,Any}}}, ::JLD.JldFile, ::Ptr{UInt8}) at /Users/monty/.julia/packages/JLD/1BoSz/src/jld_types.jl:434 [11] read_scalar(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Type) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:398 [12] read(::JLD.JldDataset) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:370 [13] read_ref(::JLD.JldFile, ::HDF5.HDF5ReferenceObj) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:502 [14] jlconvert(::Type{getfield(JLD, Symbol("##TensorDecompositions.Tucker{Core.Float64,3}#21182"))}, ::JLD.JldFile, ::Ptr{UInt8}) at /Users/monty/.julia/packages/JLD/1BoSz/src/jld_types.jl:434 [15] read_scalar(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Type) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:398 [16] read(::JLD.JldDataset) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:370 [17] read_ref(::JLD.JldFile, ::HDF5.HDF5ReferenceObj) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:502 [18] read_refs(::JLD.JldDataset, ::Type{Any}, ::Int32, ::Int32, ::Tuple{Int64}) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:475 [19] read_array(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Int32, ::Int32, ::Tuple{Int64}) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:409 [20] read_array(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Int32, ::Int32) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:408 [21] read(::JLD.JldDataset) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:372 [22] read(::JLD.JldFile, ::String) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:346 [23] #42 at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:1240 [inlined] [24] #jldopen#14(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::getfield(JLD, Symbol("##42#43")){String}, ::String, ::Vararg{String,N} where N) at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:246 [25] jldopen at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:244 [inlined] [26] load at /Users/monty/.julia/packages/JLD/1BoSz/src/JLD.jl:1239 [inlined] [27] #load#13(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::String, ::String) at /Users/monty/.julia/packages/FileIO/YJO7Z/src/loadsave.jl:118 [28] load at /Users/monty/.julia/packages/FileIO/YJO7Z/src/loadsave.jl:118 [inlined]

loleg commented 5 years ago

+1 - same issue when saving an array loaded from JSON (containing Dicts, strings, ints), working inside of IJulia with Julia 1.0.3

githubtomtom commented 5 years ago

+1 same issue: "ERROR: unrecognized HDF5 datatype class 4" loading an object of my defined type.

rasmushenningsson commented 5 years ago

I get the same error in both windows 10 and ubuntu (using virtualbox) when running the Unit Tests for JLD. It fails on Bools (see #251).

The good news is that the bug does not appear for me on master. Could someone make a new release?

hronellenfitsch commented 5 years ago

I am having the same issue, with a jld file containing Dicts. Julia 1.1.1 and latest JLD 0.9.1.

Azenor commented 5 years ago

I get the same error, I ran some simulations on a cluster and saved them in a jld file, containing Dicts as well. When trying to open the file either back on my computer (ubuntu, julia v1.1.0) or directly on the cluster I get this issue. Any solutions so far? Does it do the same with JLD2?

hronellenfitsch commented 5 years ago

@Azenor I couldn't get JLD to work, but had no trouble with BSON.jl. That might be a good alternative until JLD is fixed.

Azenor commented 5 years ago

thank you @hronellenfitsch ! I tried with BSON but my object is an array of Dicts actually so it did not work, I used JLD2 in stead it seems to work (just got this warning due to the type of one of my variables : "Warning: type LinearAlgebra.Adjoint{Float64,Array{Float64,2}} does not exist in workspace; reconstructing").

iakovost commented 4 years ago

Do we have an update on this issue?

Sinansi commented 4 years ago

This error is headache. I have lost 21 hours of work due to this error. So what is causing it? Is it due to saving a variable of boolean type?

timholy commented 4 years ago

Someone needs to supply a reproducible test case, otherwise it's essentially unfixable.

Sinansi commented 4 years ago

Someone on stackoverflow says use JLD2 instead of JLD. He said, saving the file with extension .jld2 instead of jld solved the issue. Is JLD2 supposed to be an improved replacement of JLD? https://github.com/JuliaIO/JLD2.jl

timholy commented 4 years ago

Yes, though neither is perfect. Best practice is to report bugs (with a full reproducer) whenever you encounter a problem, so that over time whatever package you use will get better.

rasmushenningsson commented 4 years ago

Someone needs to supply a reproducible test case, otherwise it's essentially unfixable.

It is reproducible in JLD v0.9.1 - the unit tests fail. (I've seen it on multiple computers, windows 10 and ubuntu in virtualbox.)

It has been fixed on JLD master (https://github.com/JuliaIO/JLD.jl/pull/269), but no new release has been made.

ericrouviere commented 4 years ago

My problem was fixed by updating JLD on Julia 1.3.1

julia> Pkg.update("JLD")
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
 Installed JLD ─ v0.9.2
  Updating `~/.julia/environments/v1.3/Project.toml`
  [4138dd39] ↑ JLD v0.9.1 ⇒ v0.9.2
  Updating `~/.julia/environments/v1.3/Manifest.toml`
  [4138dd39] ↑ JLD v0.9.1 ⇒ v0.9.2
  [9abbd945] - Profile

Before the update saving Any arrays with bool elements caused this problem. Similar to #269 .

julia> M = Any[false, rand(1)]
2-element Array{Any,1}:
 false                    
      [0.7697653460076719]

julia> save("M.jld", "M", M)

julia> load("M.jld")
ERROR: unrecognized HDF5 datatype class 4
Stacktrace:
 [1] error(::String, ::Int32) at ./error.jl:42
 [2] jldatatype(::JLD.JldFile, ::HDF5.HDF5Datatype) at /home/erouviere/.julia/packages/JLD/1BoSz/src/jld_types.jl:738
 [3] read(::JLD.JldDataset) at /home/erouviere/.julia/packages/JLD/1BoSz/src/JLD.jl:370
 [4] read_ref(::JLD.JldFile, ::HDF5.HDF5ReferenceObj) at /home/erouviere/.julia/packages/JLD/1BoSz/src/JLD.jl:502
 [5] read_refs(::JLD.JldDataset, ::Type{Any}, ::Int64, ::Int64, ::Tuple{Int64}) at /home/erouviere/.julia/packages/JLD/1BoSz/src/JLD.jl:475
 [6] read_array(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Int64, ::Int64, ::Tuple{Int64}) at /home/erouviere/.julia/packages/JLD/1BoSz/src/JLD.jl:409
 [7] read_array(::JLD.JldDataset, ::HDF5.HDF5Datatype, ::Int64, ::Int64) at /home/erouviere/.julia/packages/JLD/1BoSz/src/JLD.jl:408
 [8] read(::JLD.JldDataset) at /home/erouviere/.julia/packages/JLD/1BoSz/src/JLD.jl:372
 [9] read(::JLD.JldFile, ::String) at /home/erouviere/.julia/packages/JLD/1BoSz/src/JLD.jl:346
 [10] #39 at ./none:0 [inlined]
 [11] iterate at ./generator.jl:47 [inlined]
 [12] collect(::Base.Generator{Array{String,1},JLD.var"#39#41"{JLD.JldFile}}) at ./array.jl:622
 [13] (::JLD.var"#38#40")(::JLD.JldFile) at /home/erouviere/.julia/packages/JLD/1BoSz/src/JLD.jl:1234
 [14] #jldopen#14(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(jldopen), ::JLD.var"#38#40", ::String, ::Vararg{String,N} where N) at /home/erouviere/.julia/packages/JLD/1BoSz/src/JLD.jl:246
 [15] load at /home/erouviere/.julia/packages/JLD/1BoSz/src/JLD.jl:244 [inlined]
 [16] #load#13(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(load), ::String) at /home/erouviere/.julia/packages/FileIO/I1ONY/src/loadsave.jl:118
 [17] load(::String) at /home/erouviere/.julia/packages/FileIO/I1ONY/src/loadsave.jl:118
 [18] top-level scope at REPL[8]:1