oschulz / ValueShapes.jl

Duality of view between named variables and flat vectors in Julia
Other
12 stars 5 forks source link

Save `ShapedAsNT` with JLD2 #62

Open Cornelius-G opened 2 years ago

Cornelius-G commented 2 years ago

Currently, it's not possible to save ShapedAsNT objects with JLD2. Minimal example:

using ValueShapes
using JLD2

x = (a=1, b=2,)
vs = valshape(x)

x_shpd = ShapedAsNT(collect(x),vs)
jldsave("test.jld2", x = x_shpd)

This gives the following stacktrace.txt

@waldie11 and I tried solving that by implementing JLD2.writeas and Base.convert but didn't succeed.

We think the problem is somehow related to NamedTupleShape being deconstructed into its components in "JLD2/src/data/writing_datatypes.jl" finally resulting in the error in the function h5convert. This only happens if NamedTupleShape is ShapedAsNT. Even though we tried for quite some time, we couldn't find a fix.

We noticed that the problem is the same when trying to store a type with unknown size like this:

T = NTuple{N, Int} where N
jldsave("test.jld2", a = T)

(for which we are not sure if something like that should even be possible)

waldie11 commented 2 years ago

It seems to boil down to the missing support in JLD2 of flexible types in structs.

using ValueShapes
using JLD2
jldsave("test.jld2",x=NamedTupleShape{(:a,:b)})
jldsave("test.jld2",x=NamedTupleShape)
struct test{X<:(NTuple{N,Integer} where N)}
    a::X
end
jldsave("test.jld2", a=test)

struct test_workaround
    a::(NTuple{N,Integer} where N)
end
jldsave("test.jld2", a=test_workaround)

NamedTupleShapes frequently dispatches by NamedTupleShapes{names}, where N is left undefined.

Suggestion:

  1. Hope for fix in JLD2, or
  2. reduce struct
--- a/src/named_tuple_shape.jl
+++ b/src/named_tuple_shape.jl
@@ -37,14 +37,14 @@ all(x -> x == 4.2, view(flatview(data), 1, :))
 """
-struct NamedTupleShape{names,AT<:(NTuple{N,ValueAccessor} where N),VT} <: AbstractValueShape
-    _accessors::NamedTuple{names,AT}
+struct NamedTupleShape{names,VT} <: AbstractValueShape
+    _accessors::NamedTuple{names,<:NTuple{N,ValueAccessor}} where N

For our project, the latter has a long rat tail reaching into BAT.jl. However I didn't see any direct usage of the length N information or the type AT anywhere. Will check if Zygote is affected.