Closed yha closed 4 months ago
JonasIsensee
commented
4 months ago Hi @yha,
this is a problem with some erroneously converted AbstractDicts.
The problem was introduced in some 1.4.3x version and removed shortly after.
To load your files you can define the conversion:
rconvert(::Type{T}, x::Vector{ <: Pair}) where {T<:OrderedDict} = T(x)
EliLazarus
commented
4 months ago I'm getting a similar error, but only on the Github tests for my package (I added a unit test using data in a JLD2 file). Maybe it's a similar problem?
┌ Warning: type Core.Pair{Any,Any} does not exist in workspace; reconstructing
└ @ JLD2 ~/.julia/packages/JLD2/oYEEg/src/data/reconstructing_datatypes.jl:607
┌ Warning: type Array{JLD2.ReconstructedMutable{Symbol("Pair{Any,Any}"), (:first, :second), Tuple{Any, Any}},1} does not exist in workspace; reconstructing
└ @ JLD2 ~/.julia/packages/JLD2/oYEEg/src/data/reconstructing_datatypes.jl:492
Error encountered while load FileIO.File{FileIO.DataFormat{:JLD2}, String}("/home/runner/work/MPSGE.jl/MPSGE.jl/test/./gams/DAAData.jld2").
Fatal error:
ERROR: LoadError: LoadError: MethodError: Cannot `convert` an object of type JLD2.ReconstructedPrimitive{Symbol("Array{JLD2.ReconstructedMutable{Symbol(\"Pair{Any,Any}\"), (:first, :second), Tuple{Any, Any}},1}"), UInt64} to an object of type Dict{Any, Any}
Closest candidates are:
convert(::Type{T}, ::T) where T<:AbstractDict at abstractdict.jl:523
convert(::Type{T}, ::AbstractDict) where T<:AbstractDict at abstractdict.jl:525
convert(::Type{T}, ::T) where T at essentials.jl:205Because it's the automated github tests, I'm not quite sure how to change the version, and I'm not replicating the issue locally. The data is a Dict of DenseAxisArrays.
JonasIsensee
commented
4 months ago Hi @EliLazarus ,
your issue is different.
You created a file with a recent version of Julia, where the type Pair is defined in Core where as in julia 1.6 the type was still defined in Base. Therefore, JLD2 cannot find it when reconstructing.
You could avoid the problem by re-generating the file in julia 1.6.
To "just" re-save, you can do
# julia +1.6
@eval Core Pair=$(Base.Pair)
data = load("DAAData.jld2");
save("DAAData.jld2", data);(but I wouldn't recommend evalling names into Core as part of your CI...)
yha
commented
4 months ago Hi @yha, this is a problem with some erroneously converted
AbstractDicts. The problem was introduced in some 1.4.3x version and removed shortly after.
Is this referring to https://github.com/JuliaIO/JLD2.jl/issues/536? I tried solving my issue by sticking to version 0.4.37, to avoid having to convert a lot of older JLD2 files, but now it seems like I've hit #536 with some of those files when using this version.
To load your files you can define the conversion:
rconvert(::Type{T}, x::Vector{ <: Pair}) where {T<:OrderedDict} = T(x)
So to fix non-ordered dicts that I also have saved I need this line for general AbstractDict?
rconvert(::Type{T}, x::Vector{ <: Pair}) where {T<:AbstractDict} = T(x)?
JonasIsensee
commented
4 months ago Hi @yha, this is a problem with some erroneously converted
AbstractDicts. The problem was introduced in some 1.4.3x version and removed shortly after.Is this referring to #536? I tried solving my issue by sticking to version 0.4.37, to avoid having to convert a lot of older JLD2 files, but now it seems like I've hit #536 with some of those files when using this version.
No, this referred to #492 which defined a conversion that was applicable to too many types. This was later reverted.
So to fix non-ordered dicts that I also have saved I need this line for general
AbstractDict?rconvert(::Type{T}, x::Vector{ <: Pair}) where {T<:AbstractDict} = T(x)?
yes, that should help.
This file can't be opened with recent JLD2 versions, fixing the versions of other packages used in the file: https://www.dropbox.com/scl/fi/qih6zelvvzs1lkklb60l7/bad-jld2-file.jld2?rlkey=2bdtoexqyvcmgadv8boppod5j&dl=0
I'm not sure at this point which version of JLD2 was originally used to save it, but it was probably either 0.4.35 or 0.4.37. I have many files with similar structure, some of which fail similarly with newer JLD2 versions while other open successfully.
In a new Julia session: