search

JuliaIO / JLD2.jl

HDF5-compatible file format in pure Julia
Other
546 stars 85 forks source link

Custom serialization for SymEngine.jl required #362

Closed chelseas closed 8 months ago

chelseas commented 2 years ago

This is my error message....can any kind soul help?

julia> load("run_data/run_23903.jld2")
┌ Warning: read type Tuple{Vector{NamedTuple{(:copy_time, :sample_holds, :sample_time, :holds, :dt1, :dt2, :CE_1, :CE_2, :CE_r1, :CE_r2, :setup_plus_solve_time, :cell_time), Tuple{Float64, Bool, Float64, Bool, Float64, Float64, Dict{Any, Any}, Dict{Any, Any}, String, String, Float64, Float64}}}, Float64, String, String, Vector{Any}, Tuple, Int64, NamedTuple{(:net_encoding_time, :NL_encoding_time), Tuple{Float64, Float64}}} is not a leaf type in workspace; reconstructing
└ @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/data/reconstructing_datatypes.jl:125
┌ Warning: type parameters for NamedTuple{(:result, :result_time, :task, :target_cells, :cells, :problem, :sample_size, :info),JLD2.ReconstructedTypes.var"##Tuple{Vector{NamedTuple{(:copy_time, :sample_holds, :sample_time, :holds, :dt1, :dt2, :CE_1, :CE_2, :CE_r1, :CE_r2, :setup_plus_solve_time, :cell_time), Tuple{Float64, Bool, Float64, Bool, Float64, Float64, Dict{Any, Any}, Dict{Any, Any}, String, String, Float64, Float64}}}, Float64, String, String, Vector{Any}, Tuple, Int64, NamedTuple{(:net_encoding_time, :NL_encoding_time), Tuple{Float64, Float64}}}#257"} do not match type NamedTuple in workspace; reconstructing
└ @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/data/reconstructing_datatypes.jl:430
┌ Warning: type JuMP.VariableRef does not exist in workspace; reconstructing
└ @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/data/reconstructing_datatypes.jl:450
┌ Warning: type MathOptInterface.VariableIndex does not exist in workspace; reconstructing
└ @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/data/reconstructing_datatypes.jl:450
Error encountered while load FileIO.File{FileIO.DataFormat{:JLD2}, String}("run_data/run_23903.jld2").

Fatal error:
ERROR: TypeError: in Union, expected Type, got a value of type JLD2.UnknownType{String}
Stacktrace:
  [1] jlconvert(#unused#::JLD2.ReadRepresentation{Union, JLD2.OnDiskRepresentation{(0, 16, 32), Tuple{String, Vector{Any}, Vector{Any}}, Tuple{JLD2.Vlen{String}, JLD2.Vlen{JLD2.RelOffset}, JLD2.Vlen{JLD2.RelOffset}}}()}, f::JLD2.JLDFile{JLD2.MmapIO}, ptr::Ptr{Nothing}, header_offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/data/writing_datatypes.jl:479
  [2] read_scalar
    @ ~/.julia/packages/JLD2/DcnTD/src/dataio.jl:37 [inlined]
  [3] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, rr::JLD2.ReadRepresentation{Union, JLD2.OnDiskRepresentation{(0, 16, 32), Tuple{String, Vector{Any}, Vector{Any}}, Tuple{JLD2.Vlen{String}, JLD2.Vlen{JLD2.RelOffset}, JLD2.Vlen{JLD2.RelOffset}}}()}, read_dataspace::Tuple{JLD2.ReadDataspace, JLD2.RelOffset, Int64, UInt16}, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:170
  [4] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, dataspace::JLD2.ReadDataspace, datatype_class::UInt8, datatype_offset::Int64, data_offset::Int64, data_length::Int64, filter_id::UInt16, header_offset::JLD2.RelOffset, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:149
  [5] load_dataset(f::JLD2.JLDFile{JLD2.MmapIO}, offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:92
  [6] types_from_refs(f::JLD2.JLDFile{JLD2.MmapIO}, ptr::Ptr{Nothing})
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/data/reconstructing_datatypes.jl:295
  [7] jlconvert(rr::JLD2.ReadRepresentation{DataType, JLD2.OnDiskRepresentation{(0, 16), Tuple{String, Vector{Any}}, Tuple{JLD2.Vlen{String}, JLD2.Vlen{JLD2.RelOffset}}}()}, f::JLD2.JLDFile{JLD2.MmapIO}, ptr::Ptr{Nothing}, header_offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/data/reconstructing_datatypes.jl:311
  [8] read_scalar
    @ ~/.julia/packages/JLD2/DcnTD/src/dataio.jl:37 [inlined]
  [9] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, rr::JLD2.ReadRepresentation{DataType, JLD2.OnDiskRepresentation{(0, 16), Tuple{String, Vector{Any}}, Tuple{JLD2.Vlen{String}, JLD2.Vlen{JLD2.RelOffset}}}()}, read_dataspace::Tuple{JLD2.ReadDataspace, JLD2.RelOffset, Int64, UInt16}, attributes::Nothing)
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:170
 [10] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, dataspace::JLD2.ReadDataspace, datatype_class::UInt8, datatype_offset::Int64, data_offset::Int64, data_length::Int64, filter_id::UInt16, header_offset::JLD2.RelOffset, attributes::Nothing)
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:149
 [11] read_data (repeats 3 times)
    @ ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:144 [inlined]
 [12] read_attr_data(f::JLD2.JLDFile{JLD2.MmapIO}, attr::JLD2.ReadAttribute)
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:102
 [13] jltype(f::JLD2.JLDFile{JLD2.MmapIO}, cdt::JLD2.CommittedDatatype)
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/data/reconstructing_datatypes.jl:56
--- the last 8 lines are repeated 2 more times ---
 [30] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, dataspace::JLD2.ReadDataspace, datatype_class::UInt8, datatype_offset::Int64, data_offset::Int64, data_length::Int64, filter_id::UInt16, header_offset::JLD2.RelOffset, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:146
 [31] load_dataset(f::JLD2.JLDFile{JLD2.MmapIO}, offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:92
 [32] jlconvert
    @ ~/.julia/packages/JLD2/DcnTD/src/data/writing_datatypes.jl:301 [inlined]
 [33] macro expansion
    @ ~/.julia/packages/JLD2/DcnTD/src/dataio.jl:70 [inlined]
 [34] macro expansion
    @ ./simdloop.jl:77 [inlined]
 [35] read_array!
    @ ~/.julia/packages/JLD2/DcnTD/src/dataio.jl:68 [inlined]
 [36] read_array(f::JLD2.JLDFile{JLD2.MmapIO}, dataspace::JLD2.ReadDataspace, rr::JLD2.ReadRepresentation{Any, JLD2.RelOffset}, data_length::Int64, filter_id::UInt16, header_offset::JLD2.RelOffset, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:326
 [37] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, rr::JLD2.ReadRepresentation{Any, JLD2.RelOffset}, read_dataspace::Tuple{JLD2.ReadDataspace, JLD2.RelOffset, Int64, UInt16}, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:200
 [38] macro expansion
    @ ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:156 [inlined]
 [39] macro expansion
    @ ~/.julia/packages/JLD2/DcnTD/src/datatypes.jl:76 [inlined]
 [40] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, dataspace::JLD2.ReadDataspace, datatype_class::UInt8, datatype_offset::Int64, data_offset::Int64, data_length::Int64, filter_id::UInt16, header_offset::JLD2.RelOffset, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:152
 [41] load_dataset(f::JLD2.JLDFile{JLD2.MmapIO}, offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:92
 [42] jlconvert
    @ ~/.julia/packages/JLD2/DcnTD/src/data/writing_datatypes.jl:301 [inlined]
 [43] macro expansion
    @ ~/.julia/packages/JLD2/DcnTD/src/data/reconstructing_datatypes.jl:557 [inlined]
 [44] jlconvert(#unused#::JLD2.ReadRepresentation{JLD2.ReconstructedTypes.var"##NamedTuple{(:result, :result_time, :task, :target_cells, :cells, :problem, :sample_size, :info),JLD2.ReconstructedTypes.var\"##Tuple{Vector{NamedTuple{(:copy_time, :sample_holds, :sample_time, :holds, :dt1, :dt2, :CE_1, :CE_2, :CE_r1, :CE_r2, :setup_plus_solve_time, :cell_time), Tuple{Float64, Bool, Float64, Bool, Float64, Float64, Dict{Any, Any}, Dict{Any, Any}, String, String, Float64, Float64}}}, Float64, String, String, Vector{Any}, Tuple, Int64, NamedTuple{(:net_encoding_time, :NL_encoding_time), Tuple{Float64, Float64}}}#257\"}#258", JLD2.OnDiskRepresentation{(0, 8, 16, 32, 48, 56, 208, 216), Tuple{Any, Float64, String, String, Any, Tuple, Int64, NamedTuple{(:net_encoding_time, :NL_encoding_time), Tuple{Float64, Float64}}}, Tuple{JLD2.RelOffset, Float64, JLD2.Vlen{String}, JLD2.Vlen{String}, JLD2.RelOffset, JLD2.OnDiskRepresentation{(0, 8, 16, 24, 32, 40, 48, 56, 104, 112, 120, 128, 136, 144), Tuple{Dict{Symbol, Vector{Float64}}, Float64, Dict{String, String}, Float64, Float64, Any, Any, Tuple{Tuple{Int64, Int64}, Tuple{Int64, Int64}, Tuple{Int64, Int64}}, Any, Dict{Symbol, Symbol}, Any, Any, Float64, Any}, Tuple{JLD2.CustomSerialization{Array, JLD2.RelOffset}, Float64, JLD2.CustomSerialization{Array, JLD2.RelOffset}, Float64, Float64, JLD2.RelOffset, JLD2.RelOffset, Tuple{Tuple{Int64, Int64}, Tuple{Int64, Int64}, Tuple{Int64, Int64}}, JLD2.RelOffset, JLD2.CustomSerialization{Array, JLD2.RelOffset}, JLD2.RelOffset, JLD2.RelOffset, Float64, JLD2.RelOffset}}(), Int64, NamedTuple{(:net_encoding_time, :NL_encoding_time), Tuple{Float64, Float64}}}}()}, f::JLD2.JLDFile{JLD2.MmapIO}, ptr::Ptr{Nothing}, header_offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/data/reconstructing_datatypes.jl:502
 [45] read_scalar
    @ ~/.julia/packages/JLD2/DcnTD/src/dataio.jl:37 [inlined]
 [46] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, rr::JLD2.ReadRepresentation{JLD2.ReconstructedTypes.var"##NamedTuple{(:result, :result_time, :task, :target_cells, :cells, :problem, :sample_size, :info),JLD2.ReconstructedTypes.var\"##Tuple{Vector{NamedTuple{(:copy_time, :sample_holds, :sample_time, :holds, :dt1, :dt2, :CE_1, :CE_2, :CE_r1, :CE_r2, :setup_plus_solve_time, :cell_time), Tuple{Float64, Bool, Float64, Bool, Float64, Float64, Dict{Any, Any}, Dict{Any, Any}, String, String, Float64, Float64}}}, Float64, String, String, Vector{Any}, Tuple, Int64, NamedTuple{(:net_encoding_time, :NL_encoding_time), Tuple{Float64, Float64}}}#257\"}#258", JLD2.OnDiskRepresentation{(0, 8, 16, 32, 48, 56, 208, 216), Tuple{Any, Float64, String, String, Any, Tuple, Int64, NamedTuple{(:net_encoding_time, :NL_encoding_time), Tuple{Float64, Float64}}}, Tuple{JLD2.RelOffset, Float64, JLD2.Vlen{String}, JLD2.Vlen{String}, JLD2.RelOffset, JLD2.OnDiskRepresentation{(0, 8, 16, 24, 32, 40, 48, 56, 104, 112, 120, 128, 136, 144), Tuple{Dict{Symbol, Vector{Float64}}, Float64, Dict{String, String}, Float64, Float64, Any, Any, Tuple{Tuple{Int64, Int64}, Tuple{Int64, Int64}, Tuple{Int64, Int64}}, Any, Dict{Symbol, Symbol}, Any, Any, Float64, Any}, Tuple{JLD2.CustomSerialization{Array, JLD2.RelOffset}, Float64, JLD2.CustomSerialization{Array, JLD2.RelOffset}, Float64, Float64, JLD2.RelOffset, JLD2.RelOffset, Tuple{Tuple{Int64, Int64}, Tuple{Int64, Int64}, Tuple{Int64, Int64}}, JLD2.RelOffset, JLD2.CustomSerialization{Array, JLD2.RelOffset}, JLD2.RelOffset, JLD2.RelOffset, Float64, JLD2.RelOffset}}(), Int64, NamedTuple{(:net_encoding_time, :NL_encoding_time), Tuple{Float64, Float64}}}}()}, read_dataspace::Tuple{JLD2.ReadDataspace, JLD2.RelOffset, Int64, UInt16}, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:170
 [47] read_data(f::JLD2.JLDFile{JLD2.MmapIO}, dataspace::JLD2.ReadDataspace, datatype_class::UInt8, datatype_offset::Int64, data_offset::Int64, data_length::Int64, filter_id::UInt16, header_offset::JLD2.RelOffset, attributes::Vector{JLD2.ReadAttribute})
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:149
 [48] load_dataset(f::JLD2.JLDFile{JLD2.MmapIO}, offset::JLD2.RelOffset)
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/datasets.jl:92
 [49] getindex(g::JLD2.Group{JLD2.JLDFile{JLD2.MmapIO}}, name::String)
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/groups.jl:108
 [50] getindex(f::JLD2.JLDFile{JLD2.MmapIO}, name::String)
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/JLD2.jl:379
 [51] loadtodict!(d::Dict{String, Any}, g::JLD2.JLDFile{JLD2.MmapIO}, prefix::String)
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/loadsave.jl:153
 [52] loadtodict!
    @ ~/.julia/packages/JLD2/DcnTD/src/loadsave.jl:152 [inlined]
 [53] #62
    @ ~/.julia/packages/JLD2/DcnTD/src/fileio.jl:37 [inlined]
 [54] jldopen(::JLD2.var"#62#63", ::String, ::Vararg{String, N} where N; kws::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/loadsave.jl:4
 [55] jldopen
    @ ~/.julia/packages/JLD2/DcnTD/src/loadsave.jl:2 [inlined]
 [56] #fileio_load#61
    @ ~/.julia/packages/JLD2/DcnTD/src/fileio.jl:36 [inlined]
 [57] fileio_load(f::FileIO.File{FileIO.DataFormat{:JLD2}, String})
    @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/fileio.jl:36
 [58] #invokelatest#2
    @ ./essentials.jl:708 [inlined]
 [59] invokelatest
    @ ./essentials.jl:706 [inlined]
 [60] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::FileIO.Formatted; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:219
 [61] action
    @ ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:197 [inlined]
 [62] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:185
 [63] action
    @ ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:185 [inlined]
 [64] load(::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ FileIO ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:113
 [65] load(::String)
    @ FileIO ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:110
Stacktrace:
 [1] handle_error(e::TypeError, q::Base.PkgId, bt::Vector{Union{Ptr{Nothing}, Base.InterpreterIP}})
   @ FileIO ~/.julia/packages/FileIO/FUXWu/src/error_handling.jl:61
 [2] handle_exceptions(exceptions::Vector{Tuple{Any, Union{Base.PkgId, Module}, Vector{T} where T}}, action::String)
   @ FileIO ~/.julia/packages/FileIO/FUXWu/src/error_handling.jl:56
 [3] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::FileIO.Formatted; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ FileIO ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:228
 [4] action
   @ ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:197 [inlined]
 [5] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ FileIO ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:185
 [6] action
   @ ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:185 [inlined]
 [7] load(::String; options::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ FileIO ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:113
 [8] load(::String)
   @ FileIO ~/.julia/packages/FileIO/FUXWu/src/loadsave.jl:110
 [9] top-level scope
   @ REPL[3]:1
JonasIsensee commented 2 years ago

Hey, this is probably some kind of bug in the type reconstruction. However, this reconstruction is only meant as a last resort, anyway. Can you make sure to load JuMP and MathOptInterface. Data generated by those packages appears to be in the file but the packages are not loaded in your session.

┌ Warning: type JuMP.VariableRef does not exist in workspace; reconstructing
└ @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/data/reconstructing_datatypes.jl:450
┌ Warning: type MathOptInterface.VariableIndex does not exist in workspace; reconstructing
└ @ JLD2 ~/.julia/packages/JLD2/DcnTD/src/data/reconstructing_datatypes.jl:450
chelseas commented 2 years ago

Ok that does help. I have fewer error messages. I do still get this though:

julia> variable = Basic(:θ)
θ

julia> jldsave("test2.jld2"; variable)

julia> data = load("test2.jld2")
Dict{String, Any} with 1 entry:Error showing value of type Dict{String, Any}:
ERROR: Trying to print an uninitialized SymEngine Basic variable.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] toString(b::Basic)
    @ SymEngine ~/.julia/packages/SymEngine/OE2Ff/src/display.jl:5
  [3] show(io::IOContext{IOBuffer}, b::Basic)
    @ SymEngine ~/.julia/packages/SymEngine/OE2Ff/src/display.jl:13
  [4] sprint(f::Function, args::Basic; context::IOContext{Base.TTY}, sizehint::Int64)
    @ Base ./strings/io.jl:103
  [5] show(io::IOContext{Base.TTY}, #unused#::MIME{Symbol("text/plain")}, t::Dict{String, Any})
    @ Base ./show.jl:111
  [6] (::REPL.var"#38#39"{REPL.REPLDisplay{REPL.LineEditREPL}, MIME{Symbol("text/plain")}, Base.RefValue{Any}})(io::Any)
    @ REPL /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:220
  [7] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
    @ REPL /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:462
  [8] display(d::REPL.REPLDisplay, mime::MIME{Symbol("text/plain")}, x::Any)
    @ REPL /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:213
  [9] display(d::REPL.REPLDisplay, x::Any)
    @ REPL /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:225
 [10] display(x::Any)
    @ Base.Multimedia ./multimedia.jl:328
 [11] (::Media.var"#15#16"{Dict{String, Any}})()
    @ Media ~/.julia/packages/Media/ItEPc/src/compat.jl:28
 [12] hookless(f::Media.var"#15#16"{Dict{String, Any}})
    @ Media ~/.julia/packages/Media/ItEPc/src/compat.jl:14
 [13] render(#unused#::Media.NoDisplay, x::Dict{String, Any})
    @ Media ~/.julia/packages/Media/ItEPc/src/compat.jl:27
 [14] render(x::Dict{String, Any})
    @ Media ~/.julia/packages/Media/ItEPc/src/system.jl:160
 [15] display(#unused#::Media.DisplayHook, x::Dict{String, Any})
    @ Media ~/.julia/packages/Media/ItEPc/src/compat.jl:9
 [16] display(x::Any)
    @ Base.Multimedia ./multimedia.jl:328
 [17] #invokelatest#2
    @ ./essentials.jl:708 [inlined]
 [18] invokelatest
    @ ./essentials.jl:706 [inlined]
 [19] print_response(errio::IO, response::Any, show_value::Bool, have_color::Bool, specialdisplay::Union{Nothing, AbstractDisplay})
    @ REPL /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:247
 [20] (::REPL.var"#40#41"{REPL.LineEditREPL, Pair{Any, Bool}, Bool, Bool})(io::Any)
    @ REPL /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:231
 [21] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
    @ REPL /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:462
 [22] print_response(repl::REPL.AbstractREPL, response::Any, show_value::Bool, have_color::Bool)
    @ REPL /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:229
 [23] (::REPL.var"#do_respond#61"{Bool, Bool, REPL.var"#72#82"{REPL.LineEditREPL, REPL.REPLHistoryProvider}, REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
    @ REPL /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:798
 [24] #invokelatest#2
    @ ./essentials.jl:708 [inlined]
 [25] invokelatest
    @ ./essentials.jl:706 [inlined]
 [26] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:2441
 [27] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:1126
 [28] (::REPL.var"#44#49"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ./task.jl:411
JonasIsensee commented 2 years ago

Oh, I just had a look at SymEngine.jl.

SymEngine.jl is a wrapper around a C library and a SymEngine Basic variable contains an actual memory pointer. It does not make sense to store such a variable in a file, since in a new session, the target memory of such a pointer no longer exists. (JLD2 tries anyway, and recreates the variable with NULL pointers. Hence the uninitialized error)

If you don't care about the integrity of this symbolic variable / the lacking pointer, then it's all good. The above error was just related to the showing of the data. This happens after loading.

If this is not enough: Someone with sufficient knowledge of SymEngine could potentially (?) implement JLD2 specific custom serialization methods to allow storing and loading of symbolic variables. However, I am not certain, if this is feasible, as I know too little about SymEngine.

chelseas commented 2 years ago

Cool! Makes sense why it would fail. I have gotten around this error by converting the Basic variables to Expr type before saving. The conversion between this and Basic is lossless, to my knowledge. Maybe that could be the basis of a custom JLD2 save method for SymEngine Basic variables/expressions?

On Tue, Dec 14, 2021, 11:35 PM JonasIsensee @.***> wrote:

Oh, I just had a look at SymEngine.jl.

SymEngine.jl is a wrapper around a C library and a SymEngine Basic variable contains an actual memory pointer. It does not make sense to store such a variable in a file, since in a new session, the target memory of such a pointer no longer exists. (JLD2 tries anyway, and recreates the variable with NULL pointers. Hence the uninitialized error)

If you don't care about the integrity of this symbolic variable / the lacking pointer, then it's all good. The above error was just related to the showing of the data. This happens after loading.

If this is not enough: Someone with sufficient knowledge of SymEngine could potentially (?) implement JLD2 specific custom serialization methods to allow storing and loading of symbolic variables. However, I am not certain, if this is feasible, as I know too little about SymEngine.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/JuliaIO/JLD2.jl/issues/362#issuecomment-994427801, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADRQXSSU4CWMID333H36CN3URBAMJANCNFSM5KAHCPLQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

JonasIsensee commented 2 years ago

That is great to hear!

If the authors of SymEngine think this is a good idea, you could add the corresponding definitions to that library.

chelseas commented 2 years ago

Also thank you again for this knowledge that the SymEngine Basic variables wrap a memory pointer! I have been having other memory issues (segfaults) when trying to parallelize my code and knowing that this might be the culprit has helped a lot!

isuruf commented 2 years ago

@JonasIsensee, a SymEngine author here. We have a method in the C library to save a Basic expression to a set of bytes and a method to recreate the object using that set of bytes. How can that be incorporated into JLD2.jl?

JonasIsensee commented 2 years ago

Hi @isuruf ! That shouldn't be too hard to do. Please have a look at https://juliaio.github.io/JLD2.jl/dev/customserialization/

What you would need to do is to define an appropriate julia struct that contains all the relevant info (e.g. SerializedBasic), implement the conversion methods, and define JLD2.writeas(::Type{Basic}) = SerializedBasic somewhere inside SymEngine.jl (possibly via requires, if you don't want to add JLD2 as a dependency)

After that, JLD2 would automatically call the conversion methods when saving and loading.

EDIT: Actually, you may not even need a dedicated struct. I think, this might also work, if you can convert your Basic thing to a byte vector.

JLD2.writeas(::Type{Basic}) = Vector{UInt8}
Base.convert(::Type{Vector{UInt8}}, x::Basic) = convert_to_bytes(x)
Base.convert(::Type{Basic}, x::Vector{UInt8}) = convert_from_bytes(x)