Open b-phi opened 3 days ago
xref https://github.com/pandas-dev/pandas/issues/32265. See also #58988 and the comment chain https://github.com/pandas-dev/pandas/pull/58988/files#r1636855759
I wouldnt consider float64, Float64 and double[pyarrow] as the same dtype but different backends in the current state of pandas (there is a separate PDEP out there that talks about creating unified pandas dtypes)
Yes technically even going back to IEEE 757 0/0 is NaN; the Float64 data type does not consider NaN to be a missing value (i.e. NA)
What you expect is reasonable given the history of pandas, but the future of it is uncertain. @asishm has linked the proper discussion; I think in the long term the behavior of the OP is correct but we are just missing a isnan
method if you truly cared to detect NaN
Thank you for the discussion links. Some way to detect the nan beyond np.isnan(df)
would certainly cover our use case. The discrepancy between series.isna()
and series.apply(pd.isna)
is also quite confusing.
Pandas version checks
[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of pandas.
[ ] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
Issue Description
Dividing by zero with the arrow backend produces
float('nan')
which is not detected as NA by pandas when it is inside an arrow series.Expected Behavior
What is considered a NaN should not be dependent on the dtype backend used.
Installed Versions