BUG: "Python int too large" in maybe_convert_objects with numpy 1.26

bemoody commented 1 month ago

Pandas version checks

[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of pandas.
[X] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

pip install numpy==1.26.4 pandas==2.2.3

import numpy, pandas
numpy._set_promotion_state("weak_and_warn")
x = pandas.DataFrame({"x": [1]})
print(x)

Issue Description

If using numpy 1.26, and numpy is set to "weak" or "weak_and_warn" promotion mode (meant to be compatible with the behavior of numpy 2.x), this causes internal pandas functions to fail.

For example, the above command to print a trivial DataFrame results in:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/core/frame.py", line 1214, in __repr__
    return self.to_string(**repr_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/util/_decorators.py", line 333, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/core/frame.py", line 1394, in to_string
    return fmt.DataFrameRenderer(formatter).to_string(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/io/formats/format.py", line 962, in to_string
    string = string_formatter.to_string()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/io/formats/string.py", line 29, in to_string
    text = self._get_string_representation()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/io/formats/string.py", line 53, in _get_string_representation
    return self._fit_strcols_to_terminal_width(strcols)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/io/formats/string.py", line 163, in _fit_strcols_to_terminal_width
    col_lens = Series([Series(ele).str.len().max() for ele in strcols])
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/core/series.py", line 584, in __init__
    data = sanitize_array(data, index, dtype, copy)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/core/construction.py", line 654, in sanitize_array
    subarr = maybe_convert_platform(data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benjamin/v/lib/python3.11/site-packages/pandas/core/dtypes/cast.py", line 138, in maybe_convert_platform
    arr = lib.maybe_convert_objects(arr)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lib.pyx", line 2602, in pandas._libs.lib.maybe_convert_objects
OverflowError: Python int too large to convert to C long

This doesn't happen with numpy 1.26 in its default "legacy" mode. It doesn't happen with numpy 2.x in either "legacy" or "weak" mode.

More information about numpy 1.x versus 2.x and promotion modes is documented here: https://numpy.org/devdocs/numpy_2_0_migration_guide.html#changes-to-numpy-data-type-promotion

Expected Behavior

print(pandas.DataFrame({"x": [1]})) should not crash. It should work properly regardless of the global numpy promotion setting.

Installed Versions

``` INSTALLED VERSIONS ------------------ commit : 0691c5cf90477d3503834d983f69350f250a6ff7 python : 3.11.2 python-bits : 64 OS : Linux OS-release : 6.1.0-7-amd64 Version : #1 SMP PREEMPT_DYNAMIC Debian 6.1.20-2 (2023-04-08) machine : x86_64 processor : byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8 pandas : 2.2.3 numpy : 1.26.4 pytz : 2024.2 dateutil : 2.9.0.post0 pip : 23.0.1 Cython : None sphinx : None IPython : None adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : None blosc : None bottleneck : None dataframe-api-compat : None fastparquet : None fsspec : None html5lib : None hypothesis : None gcsfs : None jinja2 : None lxml.etree : None matplotlib : None numba : None numexpr : None odfpy : None openpyxl : None pandas_gbq : None psycopg2 : None pymysql : None pyarrow : None pyreadstat : None pytest : None python-calamine : None pyxlsb : None s3fs : None scipy : None sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None xlsxwriter : None zstandard : None tzdata : 2024.2 qtpy : None pyqt5 : None ```

yuanx749 commented 1 month ago

I can confirm, but it seems working on main branch.

bemoody commented 1 month ago

Thanks. But do you mean the bug does not occur with the main branch of numpy, or do you mean the bug does not occur with the main branch of pandas and version 1.26.4 of numpy?

As far as I've seen, this bug doesn't occur with the 2.x releases of numpy, only with the 1.x releases.

I tried doing this:

git clone https://github.com/pandas-dev/pandas
virtualenv v1
./v1/bin/pip install ./pandas
./v1/bin/pip install 'numpy<2'

And I also tried doing this:

virtualenv v2
./v2/bin/pip install --pre --extra-index https://pypi.anaconda.org/scientific-python-nightly-wheels/simple pandas
./v2/bin/pip install 'numpy<2'

Both installations exhibit the bug above.

If you don't see the bug, what platform/interpreter and what versions of pandas and numpy are you using?

bemoody commented 1 month ago

It seems like this is an inconsistency in numpy. Looks like "weak promotion" in 2.x doesn't apply to comparisons, but "weak promotion" in 1.x does apply to comparisons?

numpy 2.x

>>> import numpy
>>> numpy._set_promotion_state('weak')
>>> numpy.int8(1) < 1000
np.True_

numpy 1.x

>>> import numpy
>>> numpy._set_promotion_state('weak')
>>> numpy.int8(1) < 1000
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: Python integer 1000 out of bounds for int8

In pandas, this causes an exception at either line 1457:

            or (oINT64_MIN <= val < 0)

or line 2631:

                        val > oUINT64_MAX or val < oINT64_MIN):

for example:

>>> pandas._libs.lib.maybe_convert_objects(numpy.array([numpy.int64(1)], dtype=object))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lib.pyx", line 2631, in pandas._libs.lib.maybe_convert_objects
OverflowError: Python int too large to convert to C long
>>> pandas._libs.lib.maybe_convert_objects(numpy.array([numpy.uint32(1)], dtype=object))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lib.pyx", line 2628, in pandas._libs.lib.maybe_convert_objects
  File "lib.pyx", line 1457, in pandas._libs.lib.Seen.saw_int
OverflowError: Python integer -9223372036854775808 out of bounds for uint32

yuanx749 commented 1 month ago

I do not produce the bug on the main branch of pandas. My environment:

INSTALLED VERSIONS ------------------ commit : 49ca01ba9023b677f2b2d1c42e99f45595258b74 python : 3.10.14 python-bits : 64 OS : Linux OS-release : 5.10.16.3-microsoft-standard-WSL2 Version : #1 SMP Fri Apr 2 22:23:49 UTC 2021 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : C.UTF-8 LOCALE : en_US.UTF-8 pandas : 3.0.0.dev0+1580.g68d9dcab5b.dirty numpy : 1.26.4 dateutil : 2.9.0 pip : 24.2 Cython : 3.0.11 sphinx : 8.0.2 IPython : 8.27.0 adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : 4.12.3 blosc : None bottleneck : 1.4.0 fastparquet : 2024.5.0 fsspec : 2024.9.0 html5lib : 1.1 hypothesis : 6.112.1 gcsfs : 2024.9.0post1 jinja2 : 3.1.4 lxml.etree : 5.3.0 matplotlib : 3.9.2 numba : 0.60.0 numexpr : 2.10.0 odfpy : None openpyxl : 3.1.5 psycopg2 : 2.9.9 pymysql : 1.4.6 pyarrow : 17.0.0 pyreadstat : 1.2.7 pytest : 8.3.3 python-calamine : None pytz : 2024.2 pyxlsb : 1.0.10 s3fs : 2024.9.0 scipy : 1.14.1 sqlalchemy : 2.0.34 tables : 3.10.1 tabulate : 0.9.0 xarray : 2024.9.0 xlrd : 2.0.1 xlsxwriter : 3.2.0 zstandard : 0.23.0 tzdata : 2024.1 qtpy : None pyqt5 : None

pandas-dev / pandas