Custom serialization returns sometimes disk representation

koehlerson commented 2 years ago

Hey, I'm using the new functionality to store a custom serialization. I have some type, called StructuralModel, that stores everything related to a solid mechanics PDE discretization. Now I dispatched rconvert,wconvert and writeas such that I store on disk instead of my custom type a Dict because a Dict is more stable to its definition than my type that can change from time to time.

In general it works, however when I read in the StructuralModel a second time, I get instead of the StructuralModel it's Dict disk representation:

julia> file = jldopen("data/simulations/test/hyperelastic-demo/NeoHooke_λ=5.769230769230769_μ=3.846153846153846_nele=1000_quadorder=2_NewtonSolver_restol-1.0e-8_dtol-1.0e-8_linsolve-cg_hyperdemo.jld2")
JLDFile /Datadisk/repos/convexified-damage/data/simulations/test/hyperelastic-demo/NeoHooke_λ=5.769230769230769_μ=3.846153846153846_nele=1000_quadorder=2_NewtonSolver_restol-1.0e-8_dtol-1.0e-8_linsolve-cg_hyperdemo.jld2 (read-only)
 ├─🔢 model
 ├─📂 t=0.1
 │  ├─🔢 d
 │  ├─🔢 reaction
 │  └─🔢 materialstate
 ├─📂 t=0.2
 │  ├─🔢 d
 │  ├─🔢 reaction
 │  └─🔢 materialstate
 └─ ⋯ (3 more entries)

julia> file["model"]
┌ Warning: standard reconstruction of struct of type: NeumannBC
└ @ ConvexDamage /Datadisk/repos/convexified-damage/src/solver.jl:843
┌ Warning: standard reconstruction of struct of type: NewtonSolver
└ @ ConvexDamage /Datadisk/repos/convexified-damage/src/solver.jl:843
┌ Warning: standard reconstruction of struct of type: Grid
└ @ ConvexDamage /Datadisk/repos/convexified-damage/src/solver.jl:843
FE Model

julia> file["model"]
Dict{String, Any} with 28 entries:
  "decomposition_structasdict" => Dict{String, Any}("material"=>Dict{String, DataType}("material"=>Materi…
  "cv"                         => CellVectorValues{3, Float64, RefCube, 9}(Vec{3, Float64}[[0.490563, 0.0…
  "nbcs"                       => NeumannBC{3, Function}(["top", "bottom", "front", "back"], Function[neu…
  "savetime"                   => NthSaveTime(1, 1)
  "ip"                         => Lagrange{3, RefCube, 1}()
  "qr_ele"                     => QuadratureRule{3, RefCube, Float64}([1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0,…
  "stringtag"                  => "hyperdemo"
  "bodyforces"                 => [0.0, -0.5, 0.0]
  "compressjld2"               => true
  "reactionhistory_save"       => true
  "fv"                         => FaceVectorValues{3, Float64, RefCube, 9}(Vec{3, Float64}[[0.25, 0.0, 0.…
  "material"                   => NeoHooke[NeoHooke(3.84615, 5.76923)]
  "jld2_save"                  => true
  "λ"                          => 0.1:0.1:0.5
  "reconstinfo"                => Dict{String, Any}("ip_type"=>Lagrange, "quad_order_ele"=>2, "dh_fielddi…
  "solver"                     => NewtonSolver{typeof(cg), NoLineSearch}(5, 30, 20, 0.1, 1.0e-11, 1.0e-8,…
  "qr_face"                    => QuadratureRule{2, RefCube, Float64}([4.0], Vec{2, Float64}[[0.0, 0.0]])
  "quad_order_ele"             => 2
  "dh"                         => DofHandler{3, Float64, Grid{3, Hexahedron, Float64}}([:u], [3], Interpo…
  "ip_geo"                     => Lagrange{3, RefCube, 1}()
  "materialhistory_save"       => true
  "grid"                       => Grid{3, Hexahedron, Float64}(Hexahedron[Hexahedron((1, 2, 13, 12, 122, …
  "assemblystrategy"           => SerialAssemblyStrategy()
  "ref_type"                   => RefCube()
  "quad_order_face"            => 1
  "print"                      => true
  "vtk_save"                   => false
  "dbcs"                       => ConstraintHandler{DofHandler{3, Float64, Grid{3, Hexahedron, Float64}},…

Is this bug within my code or something related to the new custom serialization?

JonasIsensee commented 2 years ago

Hi @koehlerson,

this is a neat little bug you found there. Dictionaries with their hash tables and all that use CustomSerialization themselves to store the data as a Vector{Pair{K,V}}.

Two things are going wrong at the same time here: In your case, what is happening is that JLD2 seemingly cannot directly nest this CustomSerialization. This means, JLD2 will indeed attempt to actually serialize all the hash-tables. (This surprisingly works....)

When loading, you see a second error. The object is cached prior to conversion. In principle, this could probably be fixed but there are also easy workarounds. There are plausible workarounds:

Convert to something that is itself not "custom" (e.g. Vector{Any}) or Vector{Pair})

Wrap the dictionary:


julia> struct TestStruct
          x::Int
          y::Float64
          end

julia> struct SerializedTestStruct d::Dict{String,Any} end

julia> JLD2.rconvert(::Type{TestStruct}, x::SerializedTestStruct) = TestStruct(x.d["x"], x.d["y"])

julia> JLD2.wconvert(::Type{SerializedTestStruct}, x::TestStruct) = SerializedTestStruct(Dict{String,Any}("x"=>x.x, "y"=>x.y))

julia> JLD2.writeas(::Type{TestStruct}) = SerializedTestStruct

koehlerson commented 2 years ago

cool! thank you very much for the quick response. I thought about having a wrapper struct in the beginning but then I decided to remove it since it seemed unnecessary (which it isn't obviously :D). Your MWE transfered to my use case:

julia> file = jldopen("data/simulations/test/hyperelastic-demo/NeoHooke_λ=5.769230769230769_μ=3.846153846153846_nele=1000_quadorder=2_NewtonSolver_restol-1.0e-8_dtol-1.0e-8_linsolve-cg_hyperdemo.jld2")
JLDFile /Datadisk/repos/convexified-damage/data/simulations/test/hyperelastic-demo/NeoHooke_λ=5.769230769230769_μ=3.846153846153846_nele=1000_quadorder=2_NewtonSolver_restol-1.0e-8_dtol-1.0e-8_linsolve-cg_hyperdemo.jld2 (read-only)
 ├─🔢 model
 ├─📂 t=0.1
 │  ├─🔢 d
 │  ├─🔢 reaction
 │  └─🔢 materialstate
 ├─📂 t=0.2
 │  ├─🔢 d
 │  ├─🔢 reaction
 │  └─🔢 materialstate
 └─ ⋯ (3 more entries)

julia> file["model"]
┌ Warning: standard reconstruction of struct of type: NeumannBC
└ @ ConvexDamage /Datadisk/repos/convexified-damage/src/solver.jl:843
┌ Warning: standard reconstruction of struct of type: NewtonSolver
└ @ ConvexDamage /Datadisk/repos/convexified-damage/src/solver.jl:843
┌ Warning: standard reconstruction of struct of type: Grid
└ @ ConvexDamage /Datadisk/repos/convexified-damage/src/solver.jl:843
FE Model

julia> file["model"]
┌ Warning: standard reconstruction of struct of type: NeumannBC
└ @ ConvexDamage /Datadisk/repos/convexified-damage/src/solver.jl:843
┌ Warning: standard reconstruction of struct of type: NewtonSolver
└ @ ConvexDamage /Datadisk/repos/convexified-damage/src/solver.jl:843
┌ Warning: standard reconstruction of struct of type: Grid
└ @ ConvexDamage /Datadisk/repos/convexified-damage/src/solver.jl:843
FE Model

Do you think it's worth to add your explanations to the docs? I can take care of it, if you want to.

JonasIsensee commented 2 years ago

Do you think it's worth to add your explanations to the docs? I can take care of it, if you want to.

Yes, that would be great.

koehlerson commented 2 years ago

Alright, I'll add it to https://github.com/JuliaIO/JLD2.jl/pull/407

JuliaIO / JLD2.jl

Custom serialization returns sometimes disk representation #408