Closed vancleve closed 2 years ago
Hi, this has been discussed a few times before. I'd love to implement this but honestly, I just don't know how...
Yes, understand that JLD2 won't correctly save closures and that's totally fine.
But what I'm wondering is if we can have JLD2 still reliably open those files that do have saved closures. This example shows a case where JLD2 will reconstruct the object just fine (minus the closure) without the source code loaded but then balks after the source code is loaded.
I've seen this occur more in the case where I've made changes in the source since saving the .jld2 file (though not changes in the object definition so far as I can tell) and the .jld2 won't load due to this Cannot convert...to an object of type Function
error.
So is there a way to force JLD2 to load these files? Like maybe dump these fields that it cannot convert? The vector/scalar data is often what we need most, say the simulation runs, and the data it cannot convert, the closures, are the functions used to generate the data, which may already be saved in the original source code that ran the sims.
Ah, thank you. I had missed that the first time.
Yes, I thank it should be possible to return a normal ReconstructedType
instead of an error.
A more explicit way forward could be to pass a type remapping for the closure.
Running the mwe above I have the same issue (that an error is thrown) (JLD2 v0.4.31
)
I tried to get around it by doing custom serialization,
using JLD2
struct Foo{F<:Function}
fun::F
end
struct FooSerialization
fun
end
struct UndefinedFunction <:Function
fun
end
(f::UndefinedFunction)(args...; kwargs...) = throw(ErrorException("The function $(f.fun) is not defined"))
JLD2.writeas(::Type{<:Foo}) = FooSerialization
Base.convert(::Type{<:FooSerialization}, f::Foo) = FooSerialization(f.fun)
function Base.convert(::Type{<:Foo}, f::FooSerialization)
isa(f.fun, Function) && return Foo(f.fun)
return Foo(UndefinedFunction(f.fun))
end
Session 1
include("mwe.jl")
jldsave("tmp.jld2"; foo=Foo(x->x^2))
Session 2
include("mwe.jl")
d = jldopen("tmp.jld2", "r"); f=d["foo"]; close(d); f
┌ Warning: custom serialization of Foo{Main.#9#10} encountered, but the type does not exist in the workspace; the data will be read unconverted
└ @ JLD2 C:\Users\meyer\.julia\packages\JLD2\ryhNR\src\data\reconstructing_datatypes.jl:62
┌ Warning: type Main.#9#10 does not exist in workspace; reconstructing
└ @ JLD2 C:\Users\meyer\.julia\packages\JLD2\ryhNR\src\data\reconstructing_datatypes.jl:403
FooSerialization(JLD2.ReconstructedTypes.var"##Main.#9#10#312"())
I don't understand how to make it try to convert.
Calling convert(Foo, f)
after this works as expected, but is it possible to make it work during reconstruction?
Hi @KnutAM,
I'm not sure, I have a satisfactory answer available but I can explain what the problem is.
When you use CustomSerialization
, JLD2 stores three things:
FooSerialization
Foo{Main.#9#10}
for calling the correct convert
method.The last bit is the problem. It cannot do the conversion because it cannot create the datatype it needs Foo{Main.#9#10}
. (Since functions are their own weird type)
Thanks for the quick and good response @JonasIsensee !
I'm not sure if I fully understand all the checks in the code, but would it be possible to provide some overload mechanism like
JLD2.readas(::Type{<:ASerialization}) = A
to solve this problem?
And if such a readas
has been defined, call that instead of the current read_attr_data(f, julia_type_attr)
in the following place:
That is an interesting idea and it could possibly even work. You definitely found the right place to edit and I'd be interested to see if you can get it to work with that approach. (Please do open a new issue, for further comments. The above is a bit different.)
Viewing the bigger perspective, I believe it would be good to improve how JLD2 reconstructs Datatypes but in a way that keeps in in the data domain for longer before lifting it to the type domain..
MWE: