search

PumasAI / PumasPostProcessing.jl

PumasPostProcessing
Other
0 stars 0 forks source link

Serializing functions seemed infeasible #5

Closed asinghvi17 closed 4 years ago

asinghvi17 commented 4 years ago

Session 1:

julia> using JLD2, Serialization

julia> f(x) = x^2 + 1
f (generic function with 1 method)

julia> @save "f.jld2" f

Session 2 (immediately after session 1):

julia> using JLD2, Serialization

julia> @load "f.jld2" f
┌ Warning: type Main.#f does not exist in workspace; reconstructing
└ @ JLD2 ~/.julia/packages/JLD2/I2BGy/src/data.jl:1101
1-element Array{Symbol,1}:
 :f

julia> f(1)
ERROR: MethodError: objects of type JLD2.ReconstructedTypes.var"##Main.#f#253" are not callable
Stacktrace:
 [1] top-level scope at REPL[3]:1

Serialization also fails with a different error. It looks like both of the "trivial" options are not going to work.

This leaves us with the alternative of storing the source code, and materializing all possible measurements at the time we save, so that we don't run into issues with data being unavailable.

If we save the source code of the model and the parameters, could we "reconstitute" an FPM?

cc: @vjd @ChrisRackauckas @andreasnoack @logankilpatrick

ChrisRackauckas commented 4 years ago

Generic functions need to be redefined manually in new sessions. Anonymous functions should be fine.

asinghvi17 commented 4 years ago

Looks like that works...thanks @ChrisRackauckas!

@logankilpatrick I wonder if we could use JLSO, which is essentially a wrapper around Serialization with a Manifest cached as BSON, for short term storage, and materialize everything for a long term solution.

asinghvi17 commented 4 years ago

OK, so it looks like the short-term, local situation is solved. However, we do still need to figure out a long term solution for storage, which probably can't involve serialization of any sort.

andreasnoack commented 4 years ago

JLSO looks interesting. I know we discussed in on the call, but just for the record: using a serialized format for long term storage is fine as long as people in the future are okay with using old versions of Julia(Pro) to load it. The problem is that it's not portable to more recent versions of Julia and Pumas. In my opinion, it's fine that the portable solution is simply the script with the Pumas program, the data (or some way to query it), and a Manifest. I.e. you'd then have to rerun the program.