Quansight-Labs / ndindex

A Python library for manipulating indices of ndarrays
https://quansight-labs.github.io/ndindex/
MIT License
97 stars 12 forks source link

Regression in pickle and deepcopy after cythonization #191

Closed crusaderky closed 1 month ago

crusaderky commented 1 month ago

The recent cythonization work broke pickle and deepcopy. This works in ndindex 1.8 and fails in 1.9.1:

>>> import copy
>>> import ndindex
>>> copy.deepcopy(ndindex.ndindex(slice(1)))

File ~/miniforge3/envs/versioned-hdf5/lib/python3.12/copy.py:151, in deepcopy(x, memo, _nil)
    149 reductor = getattr(x, "__reduce_ex__", None)
    150 if reductor is not None:
--> 151     rv = reductor(4)
    152 else:
    153     reductor = getattr(x, "__reduce__", None)

File <stringsource>:2, in ndindex._slice._Slice.__reduce_cython__()

TypeError: no default __reduce__ due to non-trivial __cinit__

This is also visible in test_modify_metadata_fillvalue1 in versioned_hdf5:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
versioned_hdf5/replay.py:704: in modify_metadata
    recreate_dataset(f, dataset_name, newf, callback=callback)
versioned_hdf5/replay.py:92: in recreate_dataset
    dataset = callback(dataset, version_name)
versioned_hdf5/replay.py:683: in callback
    new_dataset.data_dict = deepcopy(dataset.data_dict)
../../miniforge3/envs/versioned-hdf5/lib/python3.12/copy.py:136: in deepcopy
    y = copier(x, memo)
../../miniforge3/envs/versioned-hdf5/lib/python3.12/copy.py:221: in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
../../miniforge3/envs/versioned-hdf5/lib/python3.12/copy.py:151: in deepcopy
    rv = reductor(4)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   TypeError: no default __reduce__ due to non-trivial __cinit__

CC @asmeurer

ev-br commented 1 month ago

one possible solution is to add a setstate/getstate pair: https://github.com/scipy/scipy/pull/13425/files

asmeurer commented 1 month ago

Fix at https://github.com/Quansight-Labs/ndindex/pull/192