pandas-dev / pandas

Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
https://pandas.pydata.org
BSD 3-Clause "New" or "Revised" License
42.57k stars 17.56k forks source link

BUG: 0/0 with arrow backend is not "NA" #59122

Open b-phi opened 3 days ago

b-phi commented 3 days ago

Pandas version checks

Reproducible Example

import pyarrow as pa
import pandas as pd

s = pd.Series([0], dtype=pd.ArrowDtype(pa.float64()))
(s / s).isna()

Issue Description

Dividing by zero with the arrow backend produces float('nan') which is not detected as NA by pandas when it is inside an arrow series.

In [25]: s / s
Out[25]:
0    NaN
dtype: double[pyarrow]

In [25]: (s / s).isna()
Out[25]:
0    False
dtype: bool

In [26]: (s / s).apply(pd.isna)
Out[26]:
0    True
dtype: bool

In [28]: pd.isna(float('nan'))
Out[28]: True

In [30]: pd.Series([float('nan')]).isna()
Out[30]:
0    True
dtype: bool

Expected Behavior

What is considered a NaN should not be dependent on the dtype backend used.

Installed Versions

INSTALLED VERSIONS ------------------ commit : d9cdd2ee5a58015ef6f4d15c7226110c9aab8140 python : 3.10.10.final.0 python-bits : 64 OS : Darwin OS-release : 23.5.0 Version : Darwin Kernel Version 23.5.0: Wed May 1 20:09:52 PDT 2024; root:xnu-10063.121.3~5/RELEASE_X86_64 machine : x86_64 processor : i386 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8 pandas : 2.2.2 numpy : 2.0.0 pytz : 2024.1 dateutil : 2.9.0.post0 setuptools : 70.1.1 pip : 24.1.1 Cython : None pytest : 8.2.1 hypothesis : None sphinx : 7.3.7 blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : 2.9.9 jinja2 : 3.1.4 IPython : 8.24.0 pandas_datareader : None adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : 4.12.3 bottleneck : None dataframe-api-compat : None fastparquet : None fsspec : 2024.5.0 gcsfs : None matplotlib : None numba : None numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : 16.1.0 pyreadstat : None python-calamine : None pyxlsb : None s3fs : 2024.5.0 scipy : 1.13.1 sqlalchemy : None tables : None tabulate : 0.9.0 xarray : None xlrd : None zstandard : None tzdata : 2024.1 qtpy : None pyqt5 : None
asishm commented 3 days ago

xref https://github.com/pandas-dev/pandas/issues/32265. See also #58988 and the comment chain https://github.com/pandas-dev/pandas/pull/58988/files#r1636855759

I wouldnt consider float64, Float64 and double[pyarrow] as the same dtype but different backends in the current state of pandas (there is a separate PDEP out there that talks about creating unified pandas dtypes)

WillAyd commented 11 hours ago

Yes technically even going back to IEEE 757 0/0 is NaN; the Float64 data type does not consider NaN to be a missing value (i.e. NA)

What you expect is reasonable given the history of pandas, but the future of it is uncertain. @asishm has linked the proper discussion; I think in the long term the behavior of the OP is correct but we are just missing a isnan method if you truly cared to detect NaN

b-phi commented 9 hours ago

Thank you for the discussion links. Some way to detect the nan beyond np.isnan(df) would certainly cover our use case. The discrepancy between series.isna() and series.apply(pd.isna) is also quite confusing.