ImportError: numpy.core.multiarray failed to import with NumPy 2.0.0

khaeru commented 3 weeks ago

Describe the bug Bottleneck 1.3.8 cannot be imported with NumPy 2.0.0:

>>> import bottleneck

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):  File "/home/khaeru/.venv/3.12/bin/ipython", line 8, in <module>
    sys.exit(start_ipython())
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/__init__.py", line 129, in start_ipython
    return launch_new_instance(argv=argv, **kwargs)
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/traitlets/config/application.py", line 1053, in launch_instance
    app.start()
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/terminal/ipapp.py", line 317, in start
    self.shell.mainloop()
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/terminal/interactiveshell.py", line 888, in mainloop
    self.interact()
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/terminal/interactiveshell.py", line 881, in interact
    self.run_cell(code, store_history=True)
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3046, in run_cell
    result = self._run_cell(
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3101, in _run_cell
    result = runner(coro)
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
    coro.send(None)
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3306, in run_cell_async
    has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3488, in run_ast_nodes
    if await self.run_code(code, result, async_=asy):
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3548, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-8-baae57b98b2a>", line 1, in <module>
    import bottleneck
  File "/home/khaeru/.venv/3.12/lib/python3.12/site-packages/bottleneck/__init__.py", line 7, in <module>
    from .move import (move_argmax, move_argmin, move_max, move_mean, move_median,
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
File ~/.venv/3.12/lib/python3.12/site-packages/numpy/core/_multiarray_umath.py:44, in __getattr__(attr_name)
     39     # Also print the message (with traceback).  This is because old versions
     40     # of NumPy unfortunately set up the import to replace (and hide) the
     41     # error.  The traceback shouldn't be needed, but e.g. pytest plugins
     42     # seem to swallow it and we should be failing anyway...
     43     sys.stderr.write(msg + tb_msg)
---> 44     raise ImportError(msg)
     46 ret = getattr(_multiarray_umath, attr_name, None)
     47 if ret is None:

ImportError: 
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[8], line 1
----> 1 import bottleneck

File ~/.venv/3.12/lib/python3.12/site-packages/bottleneck/__init__.py:7
      5 from . import slow
      6 from ._pytesttester import PytestTester
----> 7 from .move import (move_argmax, move_argmin, move_max, move_mean, move_median,
      8                    move_min, move_rank, move_std, move_sum, move_var)
      9 from .nonreduce import replace
     10 from .nonreduce_axis import (argpartition, nanrankdata, partition, push,
     11                              rankdata)

ImportError: numpy.core.multiarray failed to import

To Reproduce

Command/code being executed: as above.
Python version and OS: Python 3.12.3, Ubuntu 24.04.
pip version: pip 23.2.

4. Output of `pip list` or `conda list` (expand)

``` Package Version Editable project location ----------------------------- -------------------------------------- -------------------------------- accessible-pygments 0.0.4 alabaster 0.7.15 alembic 1.13.1 annotated-types 0.6.0 anyio 4.3.0 appdirs 1.4.4 argcomplete 3.3.0 argon2-cffi 23.1.0 argon2-cffi-bindings 21.2.0 arrow 1.3.0 asttokens 2.4.1 async-lru 2.0.4 asyncssh 2.14.2 attrs 23.2.0 Babel 2.14.0 beautifulsoup4 4.12.2 bleach 6.1.0 blinker 1.7.0 bottle 0.12.25 Bottleneck 1.3.8 build 1.2.1 cachelib 0.9.0 cattrs 23.2.3 certifi 2024.2.2 cffi 1.16.0 cfgv 3.4.0 charset-normalizer 3.3.2 click 8.1.7 cloudpickle 3.0.0 colorama 0.4.6 colorlog 6.8.2 comm 0.2.1 contourpy 1.2.0 coverage 7.4.0 cryptography 41.0.7 cycler 0.12.1 dask 2024.5.1 dask-expr 1.1.1 debugpy 1.8.0 decorator 5.1.1 defusedxml 0.7.1 distlib 0.3.8 docutils 0.20.1 et-xmlfile 1.1.0 execnet 2.0.2 executing 2.0.1 fastapi 0.110.1 fastjsonschema 2.19.1 feedgenerator 2.1.0 filelock 3.13.1 Flask 3.0.3 Flask-Caching 2.1.0 flexcache 0.3 flexparser 0.3.1 fonttools 4.47.0 fqdn 1.5.1 fsspec 2023.12.2 genno 1.24.2.dev32+g6b95e83.d20240323 /home/khaeru/vc/genno git-filter-repo 2.38.0 gitdb 4.0.11 GitPython 3.1.40 gprof2dot 2022.7.29 graphviz 0.20.1 greenlet 3.0.3 h11 0.14.0 h2 4.1.0 hpack 4.0.0 httpcore 1.0.5 httpx 0.27.0 hyperframe 6.0.1 iam-units 2023.9.12 icecream 2.1.3 identify 2.5.33 idna 3.7 imagesize 1.4.1 importlib-metadata 7.0.1 importlib_resources 6.4.0 iniconfig 2.0.0 ipykernel 6.28.0 ipython 8.17.1 ipywidgets 8.1.1 isodate 0.6.1 isoduration 20.11.0 itsdangerous 2.1.2 ixmp 3.8.1.dev13+g84c3817.d20240323 /home/khaeru/vc/iiasa/ixmp ixmp4 0.8.2 JayDeBeApi 1.2.3 Jinja2 3.1.2 joblib 1.3.2 JPype1 1.5.0 json5 0.9.14 jsonpointer 2.4 jsonschema 4.20.0 jsonschema-specifications 2023.12.1 jupyter 1.0.0 jupyter_client 8.6.0 jupyter-console 6.6.3 jupyter_core 5.7.0 jupyter-events 0.9.0 jupyter-lsp 2.2.1 jupyter_server 2.12.2 jupyter_server_terminals 0.5.1 jupyterlab 4.0.10 jupyterlab_pygments 0.3.0 jupyterlab_server 2.25.2 jupyterlab-widgets 3.0.9 khaeru 1.0.1.dev193+g2e4990c.d20240614 /home/khaeru/vc/khaeru kiwisolver 1.4.5 latexcodec 2.0.1 llvmlite 0.42.0 locket 1.0.0 lxml 5.1.0 lxml-stubs 0.5.1 Mako 1.3.3 markdown-it-py 3.0.0 MarkupSafe 2.1.5 matplotlib 3.8.2 matplotlib-inline 0.1.6 mdurl 0.1.2 memory-profiler 0.61.0 message_data 2020.6.21.dev3019+g184c70b00.d20240408 /home/khaeru/vc/iiasa/mdata message_ix 3.8.1.dev55+gd7c5f4f7.d20240323 /home/khaeru/vc/iiasa/message-ix message-ix-models 2024.4.3.dev11+gf08a3447 /home/khaeru/vc/iiasa/models mistune 3.0.2 mizani 0.9.3 multimethod 1.10 mypy 1.9.0 mypy-extensions 1.0.0 nbclient 0.9.0 nbconvert 7.14.0 nbformat 5.9.2 nest-asyncio 1.5.8 nodeenv 1.8.0 nomenclature-iamc 0.2 notebook 7.0.6 notebook_shim 0.2.3 nox 2024.4.15 numba 0.59.0 numpy 2.0.0 numpydoc 1.6.0 openpyxl 3.1.2 oracledb 2.1.0 ordered-set 4.1.0 overrides 7.4.0 owlready2 0.45 packaging 24.0 pandas 2.2.2 pandas-stubs 2.2.1.240316 pandera 0.18.3 pandocfilters 1.5.0 parso 0.8.3 partd 1.4.1 patsy 0.5.6 pelican 4.9.1 pexpect 4.8.0 pickleshare 0.7.5 pillow 10.2.0 Pint 0.23 pip 23.2 platformdirs 4.1.0 plotnine 0.12.4 pluggy 1.3.0 pooch 1.8.0 pprint36 3.9.16.0 pre-commit 3.6.0 pretenders 1.4.5 prometheus-client 0.19.0 prompt-toolkit 3.0.39 psutil 5.9.7 psycopg 3.1.18 psycopg-binary 3.1.18 ptyprocess 0.7.0 pure-eval 0.2.2 py-cpuinfo 9.0.0 pyam-iamc 2.2.1 pyarrow 14.0.2 pybtex 0.24.0 pybtex-docutils 1.0.3 pycountry 23.12.11 pycparser 2.21 pydantic 2.7.0 pydantic_core 2.18.1 pydantic-settings 2.2.1 pydata-sphinx-theme 0.15.1 Pygments 2.17.2 PyJWT 2.8.0 pyparsing 3.1.1 pyproject_hooks 1.0.0 pytest 8.0.0 pytest-benchmark 4.0.0 pytest-cov 4.1.0 pytest-profiling 1.7.0 pytest-rerunfailures 13.0 pytest-xdist 3.5.0 python-dateutil 2.9.0.post0 python-dotenv 1.0.1 python-json-logger 2.0.7 pytz 2024.1 pyxlsb 1.0.10 PyYAML 6.0.1 pyzmq 25.1.2 qtconsole 5.5.1 QtPy 2.4.1 rdflib 7.0.0 referencing 0.32.1 requests 2.31.0 requests-cache 1.2.0 requests-mock 1.11.0 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rich 13.7.1 rpds-py 0.16.2 scikit-learn 1.3.2 scipy 1.11.4 sdmx1 2.14.1.dev14 /home/khaeru/vc/sdmx seaborn 0.13.1 Send2Trash 1.8.2 setuptools 69.0.3 setuptools-scm 8.0.4 shellingham 1.5.4 six 1.16.0 smmap 5.0.1 sniffio 1.3.1 snowballstemmer 2.2.0 soupsieve 2.5 sparse 0.15.1 Sphinx 7.2.6 sphinx-book-theme 1.1.0 sphinx-rtd-theme 2.0.0 sphinxcontrib-applehelp 1.0.7 sphinxcontrib-bibtex 2.6.1 sphinxcontrib-devhelp 1.0.5 sphinxcontrib-htmlhelp 2.0.4 sphinxcontrib-jquery 4.1 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.6 sphinxcontrib-serializinghtml 1.1.9 SQLAlchemy 2.0.29 SQLAlchemy-Utils 0.41.2 stack-data 0.6.3 starlette 0.37.2 statsmodels 0.14.1 tabulate 0.9.0 terminado 0.18.0 threadpoolctl 3.2.0 tinycss2 1.2.1 toml 0.10.2 toolz 0.12.0 tornado 6.4 tqdm 4.66.2 traitlets 5.13.0 transport-energy 2021.5.5.dev45+ga7d092b /home/khaeru/vc/item/db typeguard 4.2.1 typer 0.12.3 types-docutils 0.20.0.20240317 types-python-dateutil 2.8.19.20240106 types-pytz 2024.1.0.20240203 types-PyYAML 6.0.12.20240311 types-requests 2.31.0.20240311 types-tqdm 4.66.0.20240106 typing_extensions 4.11.0 typing-inspect 0.9.0 tzdata 2024.1 Unidecode 1.3.8 uri-template 1.3.0 url-normalize 1.4.3 urllib3 2.1.0 uvicorn 0.29.0 virtualenv 20.25.0 watchfiles 0.22.0 wcwidth 0.2.9 webcolors 1.13 webencodings 0.5.1 websocket-client 1.7.0 Werkzeug 3.0.2 widgetsnbextension 4.0.9 wquantiles 0.6 wrapt 1.16.0 xarray 2023.12.0 xdg-base-dirs 6.0.1 xlrd 2.0.1 XlsxWriter 3.2.0 zipp 3.17.0 ```

Expected behavior bottleneck is importable and usable.

Additional context Encountered via daily CI for the genno package, here.

neutrinoceros commented 3 weeks ago

@khaeru in case you didn't know: bottleneck 1.4.0rc5 is compatible, so you may be able to hotfix your CI by requiring the pre-release somehow.

khaeru commented 3 weeks ago

bottleneck 1.4.0rc5 is compatible, so you may be able to hotfix your CI by requiring the pre-release somehow.

Yes, thank you; I did see it at https://pypi.org/project/Bottleneck/#history and should have come back to mention for others in the same boat.

In our case:

I'm not familiar with the maintenance practices of bottleneck.
Thus, I don't know if the >3 week old RC means there might perhaps still be other release-blocking issues in 1.4.0, and don't have the bandwidth to discover this. (The release notes don't even mention 1.3.8.)
We have other dependencies that also have NumPy 2.0 issues.

…so the simpler expedient is to temporarily limit to numpy < 2.

seberg commented 3 weeks ago

It would indeed be great to expedite a release, even if it might still have some issues. E.g. if users do a fresh install with pandas and bottleneck they otherwise run into issues (already at import time, although if not then it would happen later I guess).

rdbisme commented 3 weeks ago

Hello, I'm working on the release.

Thus, I don't know if the >3 week old RC means there might perhaps still be other release-blocking issues in 1.4.0, and don't have the bandwidth to discover this. (The release notes don't even mention 1.3.8.)

Unfortunately, I have no access to RTD. I tried to ask, but nobody gave me access. The RELEASE.rst file was up to date tho in the repo.

seberg commented 3 weeks ago

Thanks for getting the release on PyPI!

khaeru commented 3 weeks ago

Ditto, thanks!

Unfortunately, I have no access to RTD. I tried to ask, but nobody gave me access.

That's unfortunate indeed…my impression had been that Pydata were better organized than that 🙁

Perhaps until they resolve that access, the README could be updated to say “(up to v1.3.7 only)” right next to the RTD link.

pydata / bottleneck

ImportError: numpy.core.multiarray failed to import with NumPy 2.0.0 #453