search

JuliaIO / JLD2.jl

HDF5-compatible file format in pure Julia
Other
550 stars 85 forks source link

error with dynamically loaded CodecZlib #368

Closed kunyuan closed 2 years ago

kunyuan commented 2 years ago

I have some problems with compressed data saving using JLD2.jl. I find the following simple script will go wrong if the CodecZlib is dynamically loaded.

using JLD2
# using CodecZlib

mutable struct A
    m::Matrix{Float64}
    n::Matrix{Float64}
end

N = 100
a = A(rand(N, N), rand(N, N))
save("test2.jld2", Dict("a" => a), compress = true)

I got the following error message. 

❯ julia test.jl
[ Info: Attempting to dynamically load CodecZlib
Error encountered while save FileIO.File{FileIO.DataFormat{:JLD2}, String}("test2.jld2").

Fatal error:
ERROR: LoadError: MethodError: no method matching CodecZlib.ZlibCompressor()
The applicable method may be too new: running in world age 29635, while current world is 29637.
Closest candidates are:
  CodecZlib.ZlibCompressor(; level, windowbits) at /Users/kunchen/.julia/packages/CodecZlib/ruMLE/src/compression.jl:72 (method too new to be called from this world context.)
  CodecZlib.ZlibCompressor(::CodecZlib.ZStream, ::Int64, ::Int64) at /Users/kunchen/.julia/packages/CodecZlib/ruMLE/src/compression.jl:57 (method too new to be called from this world context.)
  CodecZlib.ZlibCompressor(::Any, ::Any, ::Any) at /Users/kunchen/.julia/packages/CodecZlib/ruMLE/src/compression.jl:57 (method too new to be called from this world context.)
Stacktrace:
  [1] get_compressor(#unused#::Bool)
    @ JLD2 ~/project/JLD2.jl/src/compression.jl:99
  [2] write_dataset(f::JLD2.JLDFile{JLD2.MmapIO}, dataspace::JLD2.WriteDataspace{2, Tuple{}}, datatype::JLD2.FloatingPointDatatype, odr::Type{Float64}, data::Matrix{Float64}, wsession::JLD2.JLDWriteSession{Dict{UInt64, JLD2.RelOffset}}, compress::Bool)
    @ JLD2 ~/project/JLD2.jl/src/datasets.jl:367
  [3] write_dataset(f::JLD2.JLDFile{JLD2.MmapIO}, x::Matrix{Float64}, wsession::JLD2.JLDWriteSession{Dict{UInt64, JLD2.RelOffset}}, compress::Bool)
    @ JLD2 ~/project/JLD2.jl/src/inlineunion.jl:44
  [4] write_dataset(f::JLD2.JLDFile{JLD2.MmapIO}, x::Matrix{Float64}, wsession::JLD2.JLDWriteSession{Dict{UInt64, JLD2.RelOffset}})
    @ JLD2 ~/project/JLD2.jl/src/inlineunion.jl:36
  [5] write_ref_mutable
    @ ~/project/JLD2.jl/src/datasets.jl:522 [inlined]
  [6] write_ref
    @ ~/project/JLD2.jl/src/datasets.jl:530 [inlined]
  [7] h5convert!
    @ ~/project/JLD2.jl/src/data/writing_datatypes.jl:285 [inlined]
  [8] macro expansion
    @ ~/project/JLD2.jl/src/data/writing_datatypes.jl:224 [inlined]
  [9] h5convert!(out::JLD2.IndirectPointer, #unused#::JLD2.OnDiskRepresentation{(0, 8), Tuple{Matrix{Float64}, Matrix{Float64}}, Tuple{JLD2.RelOffset, JLD2.RelOffset}}, file::JLD2.JLDFile{JLD2.MmapIO}, x::A, wsession::JLD2.JLDWriteSession{Dict{UInt64, JLD2.RelOffset}})
    @ JLD2 ~/project/JLD2.jl/src/data/writing_datatypes.jl:224
 [10] write_data(io::JLD2.MmapIO, f::JLD2.JLDFile{JLD2.MmapIO}, data::A, odr::JLD2.OnDiskRepresentation{(0, 8), Tuple{Matrix{Float64}, Matrix{Float64}}, Tuple{JLD2.RelOffset, JLD2.RelOffset}}, #unused#::JLD2.HasReferences, wsession::JLD2.JLDWriteSession{Dict{UInt64, JLD2.RelOffset}})
    @ JLD2 ~/project/JLD2.jl/src/dataio.jl:96
 [11] write_dataset(f::JLD2.JLDFile{JLD2.MmapIO}, dataspace::JLD2.WriteDataspace{0, Tuple{}}, datatype::JLD2.CommittedDatatype, odr::JLD2.OnDiskRepresentation{(0, 8), Tuple{Matrix{Float64}, Matrix{Float64}}, Tuple{JLD2.RelOffset, JLD2.RelOffset}}, data::A, wsession::JLD2.JLDWriteSession{Dict{UInt64, JLD2.RelOffset}})
    @ JLD2 ~/project/JLD2.jl/src/datasets.jl:449
 [12] write_dataset
    @ ~/project/JLD2.jl/src/datasets.jl:517 [inlined]
 [13] write(g::JLD2.Group{JLD2.JLDFile{JLD2.MmapIO}}, name::String, obj::A, wsession::JLD2.JLDWriteSession{Dict{UInt64, JLD2.RelOffset}}; compress::Nothing)
    @ JLD2 ~/project/JLD2.jl/src/compression.jl:87
 [14] write(f::JLD2.JLDFile{JLD2.MmapIO}, name::String, obj::A, wsession::JLD2.JLDWriteSession{Dict{UInt64, JLD2.RelOffset}}; compress::Nothing)
    @ JLD2 ~/project/JLD2.jl/src/compression.jl:71
 [15] write
    @ ~/project/JLD2.jl/src/compression.jl:71 [inlined]
 [16] (::JLD2.var"#55#56"{Dict{String, A}})(file::JLD2.JLDFile{JLD2.MmapIO})
    @ JLD2 ~/project/JLD2.jl/src/fileio.jl:14
 [17] jldopen(::JLD2.var"#55#56"{Dict{String, A}}, ::String, ::Vararg{String, N} where N; kws::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:compress,), Tuple{Bool}}})
    @ JLD2 ~/project/JLD2.jl/src/loadsave.jl:4
 [18] #fileio_save#54
    @ ~/project/JLD2.jl/src/fileio.jl:6 [inlined]
 [19] invokelatest(::Any, ::Any, ::Vararg{Any, N} where N; kwargs::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:compress,), Tuple{Bool}}})
    @ Base ./essentials.jl:710
 [20] action(call::Symbol, libraries::Vector{Union{Base.PkgId, Module}}, file::FileIO.Formatted, args::Dict{String, A}; options::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:compress,), Tuple{Bool}}})
    @ FileIO ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:219
 [21] action(call::Symbol, libraries::Vector{Union{Base.PkgId, Module}}, sym::Symbol, file::String, args::Dict{String, A}; options::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:compress,), Tuple{Bool}}})
    @ FileIO ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:185
 [22] save(file::String, args::Dict{String, A}; options::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:compress,), Tuple{Bool}}})
    @ FileIO ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:129
 [23] top-level scope
    @ ~/project/Lehmann.jl/test/test.jl:11
Stacktrace:
 [1] handle_error(e::MethodError, q::Base.PkgId, bt::Vector{Union{Ptr{Nothing}, Base.InterpreterIP}})
   @ FileIO ~/.julia/packages/FileIO/FUXWu/src/error_handling.jl:61
 [2] handle_exceptions(exceptions::Vector{Tuple{Any, Union{Base.PkgId, Module}, Vector{T} where T}}, action::String)
   @ FileIO ~/.julia/packages/FileIO/FUXWu/src/error_handling.jl:56
 [3] action(call::Symbol, libraries::Vector{Union{Base.PkgId, Module}}, file::FileIO.Formatted, args::Dict{String, A}; options::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:compress,), Tuple{Bool}}})
   @ FileIO ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:228
 [4] action(call::Symbol, libraries::Vector{Union{Base.PkgId, Module}}, sym::Symbol, file::String, args::Dict{String, A}; options::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:compress,), Tuple{Bool}}})
   @ FileIO ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:185
 [5] save(file::String, args::Dict{String, A}; options::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:compress,), Tuple{Bool}}})
   @ FileIO ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:129
 [6] top-level scope
   @ ~/project/Lehmann.jl/test/test.jl:11
in expression starting at /Users/kunchen/project/Lehmann.jl/test/test.jl:11

Note that if I import CodecZlib explicitly in the test.jl script, then the above error message goes away. So the problem is related to dynamically loaded CodecZlib.

I am using julia 1.6.2 and JLD2 0.4.17, CodecZlib 0.7.0.

Do you have any idea how to fix this problem in the source code? Thanks for your help!

JonasIsensee commented 2 years ago

Hi @kunyuan , thanks for reporting this.

The problem appears to be this: https://github.com/JuliaIO/JLD2.jl/blob/eebc00865eadbfcbc1dfc970f297039af62fb68c/src/compression.jl#L94

Loading CodecZlib dynamically creates a new worldage. This is meant to be taken care of by invokelatest. However, since your struct contains two compressible arrays, there will be another call to get_compressor from the old worldage leading to the error.