Closed KnutAM closed 1 year ago
Patch coverage: 100.00
% and project coverage change: +0.49
:tada:
Comparison is base (
30dd578
) 84.04% compared to head (9411e83
) 84.53%.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.
UPDATED: There were some mistakes in the example so it wasn't really relevant.
With the new changes, the approach seems to work as expected.
Specifically, when readas
hasn't been defined by the user, there should be no change to master.
The "includeme.jl"
file:
using JLD2
# Some way to define an unknown function
struct UndefinedFunction <:Function
fun
end
(f::UndefinedFunction)(args...; kwargs...) = throw(ErrorException("The function $(f.fun) is not defined"))
# Foo has all required definitions to be reloaded back to an unknown function
struct Foo{F<:Function}
fun::F
end
struct FooSerialization
fun
end
JLD2.readas(::Type{<:FooSerialization}) = Foo
JLD2.writeas(::Type{<:Foo}) = FooSerialization
Base.convert(::Type{<:FooSerialization}, f::Foo) = FooSerialization(f.fun)
function Base.convert(::Type{<:Foo}, f::FooSerialization)
isa(f.fun, Function) && return Foo(f.fun)
return Foo(UndefinedFunction(f.fun))
end
# We do not define JLD2.readas for Bar: It should remain unconverted
struct Bar{F<:Function}
fun::F
end
struct BarSerialization
fun
end
JLD2.writeas(::Type{<:Bar}) = BarSerialization
Base.convert(::Type{BarSerialization}, f::Bar) = BarSerialization(f.fun)
Base.convert(::Type{Bar}, f::BarSerialization) = Bar(f.fun)
function Base.convert(::Type{<:Bar}, f::BarSerialization)
isa(f.fun, Function) && return Bar(f.fun)
return Bar(UndefinedFunction(f.fun))
end
Session 1:
include("includeme.jl")
jldsave("sin.jld2"; foo=Foo(sin))
jldsave("tmp.jld2"; foo=Foo(x->x^2))
jldsave("bar.jld2"; bar=Bar(x->x^2))
Session 2:
include("includeme.jl")
s = jldopen("sin.jld2", "r"); ss=s["foo"]; close(s); ss
a = jldopen("tmp.jld2", "r"); aa=a["foo"]; close(a); aa
b = jldopen("bar.jld2", "r"); bb=b["bar"]; close(b); bb
@show ss
@show aa
@show bb
In the REPL after session 2:
julia> include("session2.jl");
┌ Warning: type Main.#2#3 does not exist in workspace; reconstructing
└ @ JLD2 C:\Users\meyer\.julia\dev\JLD2\src\data\reconstructing_datatypes.jl:411
┌ Warning: custom serialization of Bar{Main.#4#5} encountered, but the type does not exist in the workspace; the data will be read unconverted
└ @ JLD2 C:\Users\meyer\.julia\dev\JLD2\src\data\reconstructing_datatypes.jl:70
┌ Warning: type Main.#4#5 does not exist in workspace; reconstructing
└ @ JLD2 C:\Users\meyer\.julia\dev\JLD2\src\data\reconstructing_datatypes.jl:411
ss = Foo{typeof(sin)}(sin)
aa = Foo{UndefinedFunction}(UndefinedFunction(JLD2.ReconstructedTypes.var"##Main.#2#3#314"()))
bb = BarSerialization(JLD2.ReconstructedTypes.var"##Main.#4#5#315"())
this looks good to me. Thanks for your work! :)
I'll say again that this probably shouldn't be a permanent solution but it also doesn't look like it would harm anything. If you would like, you're welcome to bump the patch version, and add a changelog. If not, I'll get around to it in a few days.
Great! I'll try to add some tests and do that when I have some time.
Would you like me to also add some documentation explaining how/when to use this feature?
Great! I'll try to add some tests and do that when I have some time.
Would you like me to also add some documentation explaining how/when to use this feature?
That would be excellent! But please do put an Experimental in front ;)
I don't understand why the invalidation fails, the only diff from passing to failing is using a type-assert https://github.com/JuliaIO/JLD2.jl/compare/21cf579..18bfa02#diff-4ff9a5fb7264e7d305d989c07cc06a047e83cbcf591f75d8bb675558ea216970R59 ?
Nightly seems unrelated to me and fails on master too.
I don't understand why the invalidation fails, the only diff from passing to failing is using a type-assert https://github.com/JuliaIO/JLD2.jl/compare/21cf579..18bfa02#diff-4ff9a5fb7264e7d305d989c07cc06a047e83cbcf591f75d8bb675558ea216970R59 ?
Nightly seems unrelated to me and fails on master too.
To me, it looks more like an unrelated error in the CI run rather than an actual invalidation change. I've restarted the job.
I've restarted the job.
Great!
Not intending any of this to be updated in this pr, just a note of what is possible with this pr.
After tip from @fredrikekre I tried using Serialization from stdlib to write/save the function, and that seems to work!
With the readas
function, one can do
using Serialization
struct Foo{F<:Function}
fun::F
end
struct FooSerialization
io
end
JLD2.writeas(::Type{<:Foo}) = FooSerialization
JLD2.readas(::Type{<:FooSerialization}) = Foo
function Base.convert(::Type{<:FooSerialization}, f::Foo)
io = IOBuffer()
serialize(io, f.fun)
return FooSerialization(io)
end
function Base.convert(::Type{<:Foo}, f::FooSerialization)
seek(f.io, 0)
fun = deserialize(f.io)
return Foo(fun)
end
saving as jldsave("foo.jld2"; foo=Foo(x->x^2))
and reading gives
getfoo(file) = jldopen(file) do io; io["foo"]; end
foo=getfoo("foo.jld2")
# Output: Foo{Serialization.__deserialized_types__.var"#11#12"}(Serialization.__deserialized_types__.var"#11#12"())
foo.fun
# Output: #11 (generic function with 1 method)
foo.fun(2)
# Output: 4
Not tested a lot, but could potentially provide workarounds for some of #208, #175, #36, and #37 (And if there is a failure, it should occur during convert
in loading, so that when reopening the file this function can be redefined to handle potential issues with deserialization.
@JonasIsensee do you want any more changes before this can be merged?
I'm sorry for leaving this unanswered for so long. Was busy and forgot.
Thank you for your contribution!
Attempt for solving: https://github.com/JuliaIO/JLD2.jl/issues/288#issuecomment-1519076407
See example usage in the comment below and temporarily included script files
~It seems to work and tests pass locally, but it feels like this cannot be general.~ ~I never even had to overload the
readas
function...~