BUG: regression in master for DataFrame.sparse.from_spmatrix

[bmreiniger]

Pandas version checks

• [X] I have checked that this issue has not already been reported.

• [X] I have confirmed this bug exists on the latest version of pandas.

• [X] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

from scipy.sparse import eye
from pandas import DataFrame
DataFrame.sparse.from_spmatrix(eye(5))

Issue Description

(This bug does not exist on the latest version of pandas, but that checkbox was required to submit the issue.)

In main branch, the example produces a frame with entries 1.0 and NaN (sparse, with fill_value nan).

PR #59064 corrected a bug for other dtypes, but introduced this regression for float dtype.

Expected Behavior

In latest version (2.2.2) and earlier, produces a frame with entries 1.0 and 0.0 (sparse, with fill_value 0).

Installed Versions

INSTALLED VERSIONS ------------------ commit : 69fe98dda091cae71ea699be3f44926406851a2c python : 3.12.4 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.19045 machine : AMD64 processor : Intel64 Family 6 Model 140 Stepping 1, GenuineIntel byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : English_United States.1252 pandas : 3.0.0.dev0+1170.g69fe98dda numpy : 1.26.4 pytz : 2024.1 dateutil : 2.9.0.post0 pip : 24.0 Cython : None sphinx : None IPython : None adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : None blosc : None bottleneck : None fastparquet : None fsspec : None html5lib : None hypothesis : None gcsfs : None jinja2 : None lxml.etree : None matplotlib : 3.9.0 numba : None numexpr : None odfpy : None openpyxl : None psycopg2 : None pymysql : None pyarrow : None pyreadstat : None pytest : 8.2.2 python-calamine : None pyxlsb : None s3fs : None scipy : 1.14.0 sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None xlsxwriter : None zstandard : None tzdata : 2024.1 qtpy : None pyqt5 : None

[kushagr96]

take

[kushagr96]

The difference comes from the fact that the default fill value of numpy float64 is 0.0 whereas for pandas it is NaN. I tried changing it for the from_spmatrix function, but it introduces breaking changes for the rest of the Sparse matrix functions.

[bmreiniger]

It's my impression that scipy does everything assuming that the fill value is zero, so that when creating a dataframe from a scipy sparse array pandas should do the same. If that's the case, should this function just override the usual default pandas fill value for float type scipy sparse?

[WillAyd]

This is a tough one. We have two different ways of handling sparse values depending on the underlying type:

1. fill-with-missing-value which we do for floats, complex, datetime and timedelta
2. fill-with-zero-value which we do for ints and bools

fill-with-zero-value could work for ints, floats, and complex (and as you noted, this is how NumPy works), but that same logic would not extend to datetime types, since a 0 value there is still a valid value. On the other hand, we don't support missing values for ints and bools well (at least not using our historical NumPy types, and not without PDEP-16, so fill-with-zero-value has to be used for those

In our current state I don't see any way that we can make everyone happy. In the future, using a missing value for the sparse elements is the only way that could work across all types. So I'm inclined to say we should leave the current behavior as is since it more closely represents where I think we need to be long term, but @mroeschke @jorisvandenbossche @christopher-titchen may have thoughts as well