JuliaIO / JLD2.jl

HDF5-compatible file format in pure Julia
Other
558 stars 91 forks source link

`type does not exist in workspace; reconstructing ...` when loading interpolant from scratchspace new on 1.11 vs. 1.10 #610

Open 70Gage70 opened 1 week ago

70Gage70 commented 1 week ago

I'm not sure if this is happening because of Scratch.jl, Interpolations.jl or JLD2.jl, but the error is raised by JLD2.jl so I'm posting my MWE here. See the following little package

src ```julia module ItpScratchJLD export BAR, create_itp, load_itp using Scratch, JLD2, Interpolations struct MyStruct{I} itp::I end PATH_SCRATCH = Ref{String}() BAR = Ref{MyStruct}() function create_itp(; path = PATH_SCRATCH.x) outfile = joinpath(path, "itp.jld2") rm(outfile, force = true) itp = cubic_spline_interpolation(1:10, 1:10) |> MyStruct jldsave(outfile, itp = itp) end function load_itp(; path = PATH_SCRATCH.x) BAR.x = load(joinpath(path, "itp.jld2"), "itp") end function __init__() PATH_SCRATCH.x = @get_scratch!("TEST") try load_itp() catch nothing end end end # module ```
Project.toml ```toml name = "ItpScratchJLD" uuid = "f4526136-8e65-11ef-0aeb-f95001e491ea" authors = ["Test"] version = "0.0.1" [deps] Interpolations = "a98d9a8b-a2ab-59e6-89dd-64a1c18fca59" JLD2 = "033835bb-8acc-5ee8-8aae-3f567f8a3819" Scratch = "6c6a2e73-6563-6170-7368-637461726353" ```

This (roughly) reproduces the workflow of a much larger package, where interpolants are created and saved to a scratchspace using JLD2.jl so that they can be loaded later without having to recompute everything. If the interpolants exist, they are loaded automatically by __init__().

On 1.10.5, if you using ItpScratchJLD, then run create_itp(), then close everything, then using ItpScratchJLD again, you find that BAR is set correctly.

Screenshot 2024-10-19 at 5 23 38 PM

Now doing the same thing on 1.11.1, the warning is raised and BAR is not loaded, although it can be loaded afterwards with no issue.

Screenshot 2024-10-19 at 5 26 20 PM
WARNING MESSAGE TEXT ```toml ┌ Warning: type ItpScratchJLD.MyStruct{Interpolations.Extrapolation{Float64, 1, Interpolations.ScaledInterpolation{Float64, 1, Interpolations.BSplineInterpolation{Float64, 1, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, Interpolations.BSpline{Interpolations.Cubic{Interpolations.Line{Interpolations.OnGrid}}}, Tuple{Base.OneTo{Int64}}}, Interpolations.BSpline{Interpolations.Cubic{Interpolations.Line{Interpolations.OnGrid}}}, Tuple{UnitRange{Int64}}}, Interpolations.BSpline{Interpolations.Cubic{Interpolations.Line{Interpolations.OnGrid}}}, Interpolations.Throw{Nothing}}} does not exist in workspace; reconstructing └ @ JLD2 ~/.julia/packages/JLD2/IyyZz/src/data/reconstructing_datatypes.jl:577 ```

Very tricky.

JonasIsensee commented 1 week ago

Hi @70Gage70,

thanks for the report. I ran your examples and can reproduce. Since everything works on v1.10, and on v1.11 as well (as long as it is not executed during package loading) I don't think this is really a JLD2 issue but more on the julia side. I tested that without __init__ you can also load the data on first try even on v1.11.

Here's a further reduced problem: (remove Interpolations, make MyStruct not be parametric and just store something like an Int)

julia> using ItpScratchJLD
┌ Warning: type ItpScratchJLD.MyStruct does not exist in workspace; reconstructing

So it seems that the module ItpScratchJLD is no longer added to Main before running __init__

70Gage70 commented 1 week ago

@JonasIsensee thanks for your comments. I did a quick search on open Julia issues related to __init__, it looks like my issue is possibly a subset of this one. I confirmed that pkgdir prints nothing on 1.11.1 but the correct directory on 1.10.5.

thchr commented 6 days ago

I have a similar problem. I load some data inside __init__ which contains structs defined in the package itself. This previously worked fine (e.g., on 1.10) but on v1.11, JLD2 cannot recognize the types - even though they are defined in the package the __init__ is being called from.

thchr commented 6 days ago

Can confirm that this works on Julia v1.11.0-rc1, but not on later versions - so seems very likely to be due to the above-linked Julia issue, cf. this comment https://github.com/JuliaLang/julia/issues/56077#issuecomment-2403088414.