khaeru / genno

Efficient, transparent computation on labelled, N-dimensional data

https://genno.rtfd.io

GNU General Public License v3.0

1 stars 2 forks source link

Switch default Quantity: AttrSeries → SparseDataArray #23

Open khaeru opened 3 years ago

khaeru commented 3 years ago

Inherited from iiasa/ixmp#191:

xarray 0.13 includes support for converting pd.DataFrame to a pydata/sparse data structure. This should mostly obviate the need for the custom AttrSeries class. A PR should be opened to make the change, test performance, and make any necessary adjustments.

Resources:

Pint's technical commentary on container class hierarchy: https://pint.readthedocs.io/en/0.11/numpy.html#Technical-Commentary

AmphoraInc/xarray_mongodb, which integrates xarray, sparse, and pint, for reference (thanks @gidden).

As of genno 1.0, all code is tested with both AttrSeries and SparseDataArray to minimize surprises on switching.

27 should probably be done first.

khaeru commented 3 years ago

27 should probably be done first.

While doing #37, it turns out that sparse.COO currently lacks some methods that xarray needs to implement its basic DataArray API. These include:

numpy.flip, for xarray.bfill.
numpy.nancumprod, for DataArray.cumprod.
numpy.pad, for DataArray.shift.

These cases are now noted in the test suite, e.g.: https://github.com/khaeru/genno/blob/1ad7e5b95916ecf78ccd14cffb33decf84ea0793/genno/tests/core/test_quantity.py#L125-L128

genno is not the right place to implement them, so this issue needs to wait on those being implemented upstream.

khaeru commented 3 years ago

https://github.com/pydata/sparse/issues/1#issuecomment-792342987

khaeru commented 3 years ago

https://xarray.pydata.org/en/stable/user-guide/duckarrays.html gives a summary of missing operations.