search

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
43 stars 34 forks source link

[BUG] Python runtime failure due to incompatibe numpy #1127

Closed amahussein closed 2 weeks ago

amahussein commented 2 weeks ago

Describe the bug

This is happening after numpy 2.0 has been released causing teh following error on Python 3.9+

Traceback (most recent call last):
  File "~/.venv/bin/spark_rapids", line 5, in <module>
    from spark_rapids_tools.cmdli.tools_cli import main
  File "~/user_tools/src/spark_rapids_tools/__init__.py", line 22, in <module>
    from .utils import (
  File "~/user_tools/src/spark_rapids_tools/utils/__init__.py", line 17, in <module>
    from .util import (
  File "~/user_tools/src/spark_rapids_tools/utils/util.py", line 31, in <module>
    import pandas as pd
  File "~/.venv/lib/python3.9/site-packages/pandas/__init__.py", line 22, in <module>
    from pandas.compat import is_numpy_dev as _is_numpy_dev
  File "~/.venv/lib/python3.9/site-packages/pandas/compat/__init__.py", line 15, in <module>
    from pandas.compat.numpy import (
  File "~/.venv/lib/python3.9/site-packages/pandas/compat/numpy/__init__.py", line 4, in <module>
    from pandas.util.version import Version
  File "~/.venv/lib/python3.9/site-packages/pandas/util/__init__.py", line 1, in <module>
    from pandas.util._decorators import (  # noqa:F401
  File "~/.venv/lib/python3.9/site-packages/pandas/util/_decorators.py", line 14, in <module>
    from pandas._libs.properties import cache_readonly  # noqa:F401
  File "~/.venv/lib/python3.9/site-packages/pandas/_libs/__init__.py", line 13, in <module>
    from pandas._libs.interval import Interval
  File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
amahussein commented 2 weeks ago

A work around to work on python 3.19, and 3.10 for V24.0.6.0 is to install numpy 1.24.4