pandas-dev / pandas

Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
https://pandas.pydata.org
BSD 3-Clause "New" or "Revised" License
43.41k stars 17.83k forks source link

BUG: SparseFrameAccessor.to_dense ignores _constructor #59913

Open janezd opened 4 days ago

janezd commented 4 days ago

Pandas version checks

Reproducible Example

import scipy.sparse as sp
import pandas as pd

class MF(pd.DataFrame):
    @property
    def _constructor(self):
        return MF

arr = pd.core.arrays.SparseArray.from_spmatrix(sp.csr_matrix((3, 1)))
df = MF(arr)
print(type(df.sparse.to_dense()).__name__)

Issue Description

to_dense ignores _constructor and always returns DataFrame.

    def to_dense(self) -> DataFrame:
        from pandas import DataFrame

        data = {k: v.array.to_dense() for k, v in self._parent.items()}
        return DataFrame(data, index=self._parent.index, columns=self._parent.columns)

I think the code should look like this:

    def to_dense(self) -> DataFrame:
        data = {k: v.array.to_dense() for k, v in self._parent.items()}
        constr = self._parent._constructor
        return constr(data, index=self._parent.index, columns=self._parent.columns)

We use(d) an ugly workaround, but it will stop working in Pandas 3 because of #58733. See our temporary solution in https://github.com/biolab/orange3/pull/6897/commits/98c48e1b3b7c705f10af195a395213114dc2a917. :(

Expected Behavior

When subclassing DataFrame (which, I know, is discouraged, but sometimes difficult to avoid), to_dense should observe the constructor returned by _constructor.

Installed Versions

INSTALLED VERSIONS ------------------ commit : 23c497bb2f7e05af1fda966e7fb04db942453559 python : 3.11.10 python-bits : 64 OS : Darwin OS-release : 23.6.0 Version : Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:30 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T6000 machine : arm64 processor : arm byteorder : little LC_ALL : None LANG : en_GB.UTF-8 LOCALE : en_GB.UTF-8 pandas : 3.0.0.dev0+1524.g23c497bb2f numpy : 1.26.4 dateutil : 2.9.0.post0 pip : 24.2 Cython : 3.0.11 sphinx : 4.5.0 IPython : 8.27.0 adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : 4.13.0b2 blosc : None bottleneck : 1.4.0 fastparquet : None fsspec : None html5lib : 1.1 hypothesis : None gcsfs : None jinja2 : 3.1.4 lxml.etree : 5.3.0 matplotlib : 3.9.2 numba : None numexpr : None odfpy : None openpyxl : 3.1.5 psycopg2 : None pymysql : None pyarrow : None pyreadstat : None pytest : 8.3.3 python-calamine : None pytz : 2024.2 pyxlsb : None s3fs : None scipy : 1.15.0.dev0+git20240926.4c936c8 sqlalchemy : None tables : None tabulate : None xarray : None xlrd : 2.0.1 xlsxwriter : 3.2.0 zstandard : None tzdata : 2024.1 qtpy : 2.4.1 pyqt5 : None
rhshadrach commented 2 days ago

Thanks for the report! PRs to fix are welcome.

When subclassing DataFrame (which, I know, is discouraged, but sometimes difficult to avoid)

pandas fully supports subclassing; I would say it's only discouraged in that other solutions such as composition should be explored and only when those are found to be insufficient should subclassing be considered. Perhaps this is what you mean too.