search

JuliaIO / JLD2.jl

HDF5-compatible file format in pure Julia
Other
547 stars 85 forks source link

error reconstructing DataFrames? (with solution) #447

Closed tbeason closed 1 year ago

tbeason commented 1 year ago

I ran into an error revisiting some code that worked back in August.

I have jld files with several DataFrames. Now when I try to load them, I get

saved type DataFrames.DataFrame is missing field metadata in workspace type; reconstructing

and eventually it fails with

MethodError: Cannot `convert` an object of type JLD2.ReconstructedTypes.var"##DataFrames.DataFrame#1058" to an object of type DataFrames.DataFrame

With some hints from the docs, I defined

JLD2.rconvert(::Type{DataFrames.DataFrame}, nt::NamedTuple) = DataFrame(nt.columns,nt.colindex)

and loading using the typemap keyword work.

I am not sure why this would have been necessary? Just wanted to bring this to your attention to see if it represents a bigger problem

JonasIsensee commented 1 year ago

Hi @tbeason,

I'm glad you managed to find a solution yourself. I did something similar: https://discourse.julialang.org/t/reading-dataframes-from-jld2-files/88543/9?u=jonasisensee

The problem is that DataFrames had a breaking release that changed the struct layout of DataFrame. There is no general solution to these kinds of changes (except for package authors to provide custom serialization methods for JLD2 to use) and the only thing JLD2 can do by itself is to make the upgrade as easy as possible.

tbeason commented 1 year ago

Alright. Surprised I didn't see that thread! Closing this