search

pandas-dev / pandas

Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
https://pandas.pydata.org
BSD 3-Clause "New" or "Revised" License
43.29k stars 17.8k forks source link

BUG: Adding or multiplying a pandas nullable dtype Series with a pyarrow dtype Series raises TypeError #58602

Open jamesdow21 opened 4 months ago

jamesdow21 commented 4 months ago

Pandas version checks

Reproducible Example

import pandas as pd

a = pd.Series(range(5), dtype="Float64")
b = pd.Series(range(5), dtype="float64[pyarrow]")

a * b

Issue Description

Adding or multiplying a pandas nullable dtype Series with a pyarrow backed dtype Series raises a TypeError, but reversing the order works as expected

Tested and confirmed that the same problem occurs for any combination of numeric pandas nullable dtypes and pyarrow dtypes
('Int8', 'Int16', 'Int32', 'Int64', 'UInt8', 'UInt16', 'UInt32', 'UInt64', 'Float32', 'Float64') and ('int8[pyarrow]', 'int16[pyarrow]', 'int32[pyarrow]', 'int64[pyarrow]', 'uint8[pyarrow]', uint16[pyarrow]', 'uint32[pyarrow]', 'uint64[pyarrow]', 'float32[pyarrow]', 'float64[pyarrow]']) respectively

In [30]: a = pd.Series(range(5), dtype="Float64")
    ...: b = pd.Series(range(5), dtype="float64[pyarrow]")

In [31]: a * b
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[31], line 1
----> 1 a * b

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\common.py:76, in _unpack_zerodim_and_defer.<locals>.new_method(self, other)
     72             return NotImplemented
     74 other = item_from_zerodim(other)
---> 76 return method(self, other)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arraylike.py:202, in OpsMixin.__mul__(self, other)
    200 @unpack_zerodim_and_defer("__mul__")
    201 def __mul__(self, other):
--> 202     return self._arith_method(other, operator.mul)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\series.py:6126, in Series._arith_method(self, other, op)
   6124 def _arith_method(self, other, op):
   6125     self, other = self._align_for_op(other)
-> 6126     return base.IndexOpsMixin._arith_method(self, other, op)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\base.py:1382, in IndexOpsMixin._arith_method(self, other, op)
   1379     rvalues = np.arange(rvalues.start, rvalues.stop, rvalues.step)
   1381 with np.errstate(all="ignore"):
-> 1382     result = ops.arithmetic_op(lvalues, rvalues, op)
   1384 return self._construct_result(result, name=res_name)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\array_ops.py:273, in arithmetic_op(left, right, op)
    260 # NB: We assume that extract_array and ensure_wrapped_if_datetimelike
    261 #  have already been called on `left` and `right`,
    262 #  and `maybe_prepare_scalar_for_op` has already been called on `right`
    263 # We need to special-case datetime64/timedelta64 dtypes (e.g. because numpy
    264 # casts integer dtypes to timedelta64 when operating with timedelta64 - GH#22390)
    266 if (
    267     should_extension_dispatch(left, right)
    268     or isinstance(right, (Timedelta, BaseOffset, Timestamp))
   (...)
    271     # Timedelta/Timestamp and other custom scalars are included in the check
    272     # because numexpr will fail on it, see GH#31457
--> 273     res_values = op(left, right)
    274 else:
    275     # TODO we should handle EAs consistently and move this check before the if/else
    276     # (https://github.com/pandas-dev/pandas/issues/41165)
    277     # error: Argument 2 to "_bool_arith_check" has incompatible type
    278     # "Union[ExtensionArray, ndarray[Any, Any]]"; expected "ndarray[Any, Any]"
    279     _bool_arith_check(op, left, right)  # type: ignore[arg-type]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\common.py:76, in _unpack_zerodim_and_defer.<locals>.new_method(self, other)
     72             return NotImplemented
     74 other = item_from_zerodim(other)
---> 76 return method(self, other)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arraylike.py:202, in OpsMixin.__mul__(self, other)
    200 @unpack_zerodim_and_defer("__mul__")
    201 def __mul__(self, other):
--> 202     return self._arith_method(other, operator.mul)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\masked.py:806, in BaseMaskedArray._arith_method(self, other, op)
    803     # x ** 0 is 1.
    804     mask = np.where((self._data == 0) & ~self._mask, False, mask)
--> 806 return self._maybe_mask_result(result, mask)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\masked.py:870, in BaseMaskedArray._maybe_mask_result(self, result, mask)
    867 if result.dtype.kind == "f":
    868     from pandas.core.arrays import FloatingArray
--> 870     return FloatingArray(result, mask, copy=False)
    872 elif result.dtype.kind == "b":
    873     from pandas.core.arrays import BooleanArray

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\numeric.py:245, in NumericArray.__init__(self, values, mask, copy)
    239 if not (isinstance(values, np.ndarray) and checker(values.dtype)):
    240     descr = (
    241         "floating"
    242         if self._dtype_cls.kind == "f"  # type: ignore[comparison-overlap]
    243         else "integer"
    244     )
--> 245     raise TypeError(
    246         f"values should be {descr} numpy array. Use "
    247         "the 'pd.array' function instead"
    248     )
    249 if values.dtype == np.float16:
    250     # If we don't raise here, then accessing self.dtype would raise
    251     raise TypeError("FloatingArray does not support np.float16 dtype.")

TypeError: values should be integer numpy array. Use the 'pd.array' function instead

In [32]: b * a
Out[32]:
0     0.0
1     1.0
2     4.0
3     9.0
4    16.0
dtype: double[pyarrow]

Expected Behavior

Addition and multiplication should work with either order of operands

Installed Versions

INSTALLED VERSIONS ------------------ commit : bdc79c146c2e32f2cab629be240f01658cfb6cc2 python : 3.12.2.final.0 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.19045 machine : AMD64 processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : English_United States.1252 pandas : 2.2.1 numpy : 1.26.4 pytz : 2024.1 dateutil : 2.9.0.post0 setuptools : 69.2.0 pip : 24.0 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : 5.1.0 html5lib : None pymysql : None psycopg2 : None jinja2 : 3.1.3 IPython : 8.22.2 pandas_datareader : None adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : 4.12.3 bottleneck : 1.3.8 dataframe-api-compat : None fastparquet : None fsspec : 2024.3.1 gcsfs : None matplotlib : 3.8.3 numba : 0.59.1 numexpr : 2.9.0 odfpy : None openpyxl : None pandas_gbq : None pyarrow : 15.0.2 pyreadstat : None python-calamine : None pyxlsb : None s3fs : 2024.3.1 scipy : 1.12.0 sqlalchemy : None tables : None tabulate : None xarray : 2024.2.0 xlrd : None zstandard : None tzdata : 2024.1 qtpy : None pyqt5 : None
Aloqeely commented 4 months ago

Thanks for the report! I agree the behavior of addition and multiplication should be consistent. Addition and multiplication are commutative so a + b should be equal to b + a (even if it means both should fail to accomplish this commutativity)

PR to fix this would be welcome.

pmhatre1 commented 4 months ago

The error is ocurring in numeric.py

Screenshot 2024-05-08 at 5 14 58 PM
pmhatre1 commented 4 months ago

I believe pyarrow is not an instance of numpy array thus throwing an error. @Aloqeely