numpy version 2.0 compatibility

[GenevieveBuckley]

References and relevant issues

• Closes https://github.com/napari/napari/issues/6752
• Reference https://numpy.org/devdocs/numpy_2_0_migration_guide.html#ruff-plugin
• Previous related PR https://github.com/napari/napari/pull/2910

Description

This PR:

1. Adds a ruff linting rule to check compatibility with numpy v2.0, and
2. Updates existing napari code to ensure compatibility with numpy v2.0

[GenevieveBuckley]

Status:

• [x] Done: Added ruff linting rule for numpy 2.0 compatibility check
• [x] Done: Removed references to numpy.sctypes, which is removed in numpy v2.0
• [x] Done: Replaced np.set_string_function(None) --> np.set_printoptions() (since numpy.set_string_function is removed in numpy v2.0) EDIT: nope, this did not reset the whole thing like I expected, it only resets the formatter kwarg values. it's better to use np.set_printoptions as a context manager instead.
• [ ] Need help: How do I replace np.set_string_function(lambda arr: f'{type(arr)} {arr.shape} {arr.dtype}') with an equivalent np.set_printoptions command? This is the numpy.set_printoptions docs, and these are the old numpy.set_string_function docs. I can change the formatting of individual numpy array elements, but I can't figure out how to make it return a single summary string for the whole array.

As expected, the new ruff linting check is failing until we fix this last np.set_string_function problem:

napari/utils/_tracebacks.py:37:13: NPY201 `np.set_string_function` will be removed in NumPy 2.0. Use `np.set_printoptions` for custom printing of NumPy objects.
napari/utils/_tracebacks.py:91:17: NPY201 `np.set_string_function` will be removed in NumPy 2.0. Use `np.set_printoptions` for custom printing of NumPy objects.
napari/utils/_tracebacks.py:137:17: NPY201 `np.set_string_function` will be removed in NumPy 2.0. Use `np.set_printoptions` for custom printing of NumPy objects.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 92.41%. Comparing base (f195690) to head (0bd57fc).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #6776 +/- ## ========================================== - Coverage 92.45% 92.41% -0.04% ========================================== Files 614 614 Lines 55164 55162 -2 ========================================== - Hits 51001 50979 -22 - Misses 4163 4183 +20 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[GenevieveBuckley]

cc @JoOkuma - maybe you have a suggestion for this? It'd be nice to keep something similar to the print formatting style you introduced in https://github.com/napari/napari/pull/2910, but I'm not sure how to do that using the options available in numpy v2.0

[GenevieveBuckley]

I've looked into the numpy.set_printoptions docs some more, and I don't think we can reproduce the old behaviour exactly. But I can reduce how verbose the array is in the print output. Is this sufficient, or does anyone have a better plan?

arr = np.random.random((512,512))

Old:

<class 'numpy.ndarray'> (512, 512) float64

New (with np.printoptions(precision=5, threshold=80, edgeitems=2):)

[[0.98454 0.93896 ... 0.54017 0.85307]
 [0.48407 0.53626 ... 0.83477 0.55798]
 ...
 [0.13319 0.35231 ... 0.89343 0.93346]
 [0.15881 0.52209 ... 0.04007 0.71324]]

Default (no printoptions settings)

array([[0.98453704, 0.9389569 , 0.4197145 , ..., 0.89444065, 0.54017021,
        0.85306717],
       [0.48406665, 0.53625644, 0.58014584, ..., 0.0319012 , 0.83477052,
        0.55798271],
       [0.68847506, 0.41386651, 0.55052169, ..., 0.28323962, 0.66602634,
        0.86895102],
       ...,
       [0.21836951, 0.1869048 , 0.7513639 , ..., 0.51765584, 0.53707556,
        0.97249828],
       [0.13318953, 0.35231209, 0.7720745 , ..., 0.11958483, 0.89343037,
        0.9334581 ],
       [0.15881113, 0.52209381, 0.24165429, ..., 0.69698159, 0.04007437,
        0.7132449 ]])

[JoOkuma]

Hi @GenevieveBuckley, I couldn't find how to reproduce the old behavior in numpy 2.0.0.

There's a formatter argument, but it seems to be applied per value.

Changing the options to have a smaller output, as you did, might be a good compromise.

[JoOkuma]

This PR looks good to me.

[GenevieveBuckley]

Hi @GenevieveBuckley, I couldn't find how to reproduce the old behavior in numpy 2.0.0.

There's a formatter argument, but it seems to be applied per value.

Changing the options to have a smaller output, as you did, might be a good compromise.

Yeah, that's what I found too. It's good to hear you think that's a reasonable compromise.

[GenevieveBuckley]

Ready for review

[GenevieveBuckley]

Could we relax constraint <2 on NumPy version (line 52 in pyproject.toml) and run pre-test, or is there a known problem why we cannot test against numpy 2.0?

Sure, we can do that. I've pushed this branch to napari/napari, and set the --pre test workflow running here: https://github.com/napari/napari/actions/runs/9055845357

[GenevieveBuckley]

There's one failing job from --pre test: windows-latest py3.12 pyqt6 --pre

I've run the test a second time, same Windows access error. But windows-latest with pyqt5 passes fine, it's only windows with pyqt6 that has this problem. I'll look into it a bit more if I have some time.

  napari\_vispy\_tests\test_vispy_labels_layer.py ........Windows fatal exception: access violation
  Current thread 0x00001368 (most recent call first):
    File "D:\a\napari\napari\napari\utils\_testsupport.py", line 318 in make_napari_viewer
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\fixtures.py", line 893 in _teardown_yield_fixture
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\fixtures.py", line 1009 in finish
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\runner.py", line 544 in teardown_exact
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\runner.py", line 188 in pytest_runtest_teardown
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\runner.py", line 241 in <lambda>
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\runner.py", line 341 in from_call
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\runner.py", line 240 in call_and_report
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\runner.py", line 136 in runtestprotocol
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\runner.py", line 116 in pytest_runtest_protocol
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\main.py", line 364 in pytest_runtestloop
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\main.py", line 339 in _main
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\main.py", line 285 in wrap_session
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\main.py", line 332 in pytest_cmdline_main
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_callers.py", line 103 in _multicall
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pluggy\_hooks.py", line 513 in __call__
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\config\__init__.py", line 178 in main
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\_pytest\config\__init__.py", line 206 in console_main
    File "D:\tmp\.tox\py312-windows-pyqt6-no_cov\Lib\site-packages\pytest\__main__.py", line 7 in <module>
    File "<frozen runpy>", line 88 in _run_code
    File "<frozen runpy>", line 198 in _run_module_as_main
  py312-windows-pyqt6-no_cov: exit 3221225477 (195.62 seconds) D:\a\napari\napari> python -m pytest --color=yes --basetemp=\tmp\.tox\py312-windows-pyqt6-no_cov\tmp --ignore tools --maxfail=5 --json-report --json-report-file=D:\a\napari\napari/report-py312-windows-pyqt6-no_cov.json --save-leaked-object-graph pid=1004
  py312-windows-pyqt6-no_cov: FAIL code 3221225477 (384.62=setup[188.95]+cmd[0.05,195.62] seconds)
  evaluation failed :( (385.03 seconds)
Error: The process 'C:\hostedtoolcache\windows\Python\3.12.3\x64\python.exe' failed with exit code 4294967295
The process 'C:\hostedtoolcache\windows\Python\3.12.3\x64\python.exe' failed with exit code 4294967295

[GenevieveBuckley]

I'm not sure if the Windows access error seen above is just a flaky test. I can't reproduce the problem on my fork, and it seems to disappear here when I add the -v verbose flag to pytest in the tox.ini file. Perhaps this is not an actual problem, and we can merge as is.

[Czaki]

It is a known problem with pyqt6 and our cleaning mechanism.

[jni]

Thank you very much for getting this done @GenevieveBuckley!!! I kept seeing the NumPy announcements thinking, really should get onto that... 😅 😬 😂 So I'm super grateful!!!