search

fumitoh / modelx

Use Python like a spreadsheet!
https://modelx.io
GNU Lesser General Public License v3.0
96 stars 21 forks source link

model write: pickled pandas objects aren't backward competible, when updating pandas #34

Closed alebaran closed 3 years ago

alebaran commented 4 years ago

Not a big thing, but I have noticed that when I've upgraded pandas from 0.25.3 to 1.0.4, I wasn't able to load previously saved model versions. Object structure probably changed a bit and it wasn't loading correctly. I was able to reproduce the issue by using pickle.dump( pd.DataFrame(['a','b'],index=pd.MultiIndex.from_tuples(((1,2,3),(2,4,5)))), open( "try", "wb" ) ) with the older version and pickle.load(open(r'try', "rb")) with the newer one.

Pandas itself provides pickling function, which seems to ensure backward compatibility. The following works: pd.DataFrame(['a','b'],index=pd.MultiIndex.from_tuples(((1,2,3),(2,4,5)))).to_pickle('try') with the older version and pd.read_pickle('try') with the newer one.

It would be nice to have to use native pandas pickling.

fumitoh commented 4 years ago

read_pickle seems to have exception handling for backward compatibility https://github.com/pandas-dev/pandas/blob/v1.0.4/pandas/io/pickle.py#L187 This can't be fixed as DataFrames are not pickled individually in modelx

alebaran commented 4 years ago

I guess they also rarely change the structure

fumitoh commented 3 years ago

This was rather a critical issue so I decided to fix it.