search

pandas-dev / pandas

Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
https://pandas.pydata.org
BSD 3-Clause "New" or "Revised" License
43.71k stars 17.93k forks source link

BUG: Infinite recursion with outer merge on categorical key column if left and right dtypes are mismatching non-nullable numeric kinds #56376

Open wence- opened 11 months ago

wence- commented 11 months ago

Pandas version checks

Reproducible Example

import pandas as pd
import numpy as np
lt = pd.CategoricalDtype(categories=np.asarray([1, 2, 3], dtype="int64"))
rt = pd.CategoricalDtype(categories=np.asarray([1, 2, 3], dtype="float64"))
left = pd.DataFrame({"key": pd.Series([1, 2], dtype=lt)})
right = pd.DataFrame({"key": pd.Series([1, 3], dtype=rt)})
left.merge(right, on="key", how="outer")

Issue Description

  [...]
  File "<stdin>", line 1, in <module>
  File "/home/wence/Documents/src/rapids/third-party/pandas/pandas/core/frame.py", line 10737, in merge
    return merge(
  File "/home/wence/Documents/src/rapids/third-party/pandas/pandas/core/reshape/merge.py", line 185, in merge
    return op.get_result(copy=copy)
  File "/home/wence/Documents/src/rapids/third-party/pandas/pandas/core/reshape/merge.py", line 897, in get_result
    self._maybe_add_join_keys(result, left_indexer, right_indexer)
  File "/home/wence/Documents/src/rapids/third-party/pandas/pandas/core/reshape/merge.py", line 1076, in _maybe_add_join_keys
    key_col = Index(lvals).where(~mask_left, rvals)
  File "/home/wence/Documents/src/rapids/third-party/pandas/pandas/core/indexes/base.py", line 5237, in where
    return self.putmask(~cond, other)
  File "/home/wence/Documents/src/rapids/third-party/pandas/pandas/core/indexes/base.py", line 5493, in putmask
    return self.astype(dtype).putmask(mask, value)
  File "/home/wence/Documents/src/rapids/third-party/pandas/pandas/core/indexes/base.py", line 5493, in putmask
    return self.astype(dtype).putmask(mask, value)
  File "/home/wence/Documents/src/rapids/third-party/pandas/pandas/core/indexes/base.py", line 5493, in putmask
    return self.astype(dtype).putmask(mask, value)
  [Previous line repeated 984 more times]
  File "/home/wence/Documents/src/rapids/third-party/pandas/pandas/core/indexes/base.py", line 1057, in astype
    return self.copy() if copy else self
  File "/home/wence/Documents/src/rapids/third-party/pandas/pandas/core/indexes/base.py", line 1259, in copy
    name = self._validate_names(name=name, deep=deep)[0]
  File "/home/wence/Documents/src/rapids/third-party/pandas/pandas/core/indexes/base.py", line 1701, in _validate_names
    validate_all_hashable(*new_names, error_name=f"{type(self).__name__}.name")
  File "/home/wence/Documents/src/rapids/third-party/pandas/pandas/core/dtypes/common.py", line 1575, in validate_all_hashable
    if not all(is_hashable(arg) for arg in args):
  File "/home/wence/Documents/src/rapids/third-party/pandas/pandas/core/dtypes/common.py", line 1575, in <genexpr>
    if not all(is_hashable(arg) for arg in args):
  File "/home/wence/Documents/src/rapids/third-party/pandas/pandas/core/dtypes/inference.py", line 366, in is_hashable
    hash(obj)
RecursionError: maximum recursion depth exceeded while calling a Python object

Expected Behavior

I would expect this to complete, probably behaving as if the left and right key columns were of type int64 and float64. I would also accept raising.

Installed Versions

``` INSTALLED VERSIONS ------------------ commit : b864b8eb54605124827e6af84aa7fbc816cfc1e0 python : 3.10.12.final.0 python-bits : 64 OS : Linux OS-release : 6.2.0-37-generic Version : #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 2 18:01:13 UTC 2 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_GB.UTF-8 LOCALE : en_GB.UTF-8 pandas : 2.1.0.dev0+1414.gb864b8eb54 numpy : 1.24.4 pytz : 2023.3 dateutil : 2.8.2 setuptools : 67.7.2 pip : 23.2.1 Cython : 3.0.0 pytest : 7.4.0 hypothesis : 6.82.2 sphinx : 6.2.1 blosc : None feather : None xlsxwriter : 3.1.2 lxml.etree : 4.9.3 html5lib : 1.1 pymysql : 1.4.6 psycopg2 : 2.9.6 jinja2 : 3.1.2 IPython : 8.14.0 pandas_datareader : None adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : 4.12.2 bottleneck : 1.3.7 dataframe-api-compat : None fastparquet : 2023.7.0 fsspec : 2023.6.0 gcsfs : 2023.6.0 matplotlib : 3.7.2 numba : 0.57.1 numexpr : 2.8.4 odfpy : None openpyxl : 3.1.2 pandas_gbq : None pyarrow : 12.0.1 pyreadstat : 1.2.2 python-calamine : None pyxlsb : 1.0.10 s3fs : 2023.6.0 scipy : 1.11.1 sqlalchemy : 2.0.19 tables : 3.8.0 tabulate : 0.9.0 xarray : 2023.7.0 xlrd : 2.0.1 zstandard : 0.19.0 tzdata : 2023.3 qtpy : 2.4.0 pyqt5 : None ```
rob-sil commented 11 months ago

take

rob-sil commented 11 months ago

Looks like the same type of bug as https://github.com/pandas-dev/pandas/issues/52927, just showing up in different places.