MAINT: NumPy _ARRAY_API import deprecation warning

Required prerequisites

[X] Make sure you've read the documentation. Your issue may be addressed there.
[X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
[X] Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.11.1

Problem description

Using latest version of SciPy main along with latest version of NumPy main I see 8402 failures in the Scipy testsuite that are a bit tricky, many of which seem to be related to include/pybind11/numpy.h doing auto c = m.attr("_ARRAY_API"); and then a DeprecationWarning boils up during the SciPy testsuite.

E   DeprecationWarning: numpy.core.multiarray is deprecated and has been renamed to numpy._core.multiarray. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.multiarray._ARRAY_API.

I'll paste a full reproducer/traceback below (well, in the context of NumPy/SciPy main). I wonder if there is a simple way to suppress that particular warning in the header, since I think pybind11 is actually intending to dig into the internals there, given that there are already recent NumPy 2.x shims I see around in that file.

Or maybe I'm just missing something completely here--@rgommers or @mtsokol can yell at me if so. I think this comes through in the pybind11 bindings of pocketfft perhaps, based on the traceback.

Reproducible example code

This one is a little tricky, first apply patch from https://github.com/data-apis/array-api-compat/pull/63 to appropriate SciPy submodule on `main` branch, and when build alongside NumPy `main`:

`python dev.py test -t scipy/signal/tests/test_signaltools.py::TestOAConvolve::test_2d_axes[False-3-3-4-47-4-47-same-axes1]`

____________________________________________________________________________________ TestOAConvolve.test_2d_axes[False-3-3-4-47-4-47-same-axes1] _____________________________________________________________________________________
scipy/signal/tests/test_signaltools.py:947: in test_2d_axes
    expected = fftconvolve(a, b, mode=mode, axes=axes)
        a          = array([[[0.88156736, 0.86223916, 0.16972094, 0.89005165],
        [0.15585618, 0.624305  , 0.70602407, 0.7034736 ],
  ...,
        [0.77882918, 0.50795249, 0.24284804, 0.85650584],
        [0.35205817, 0.42588504, 0.05523513, 0.65621471]]])
        ax_a       = [4, 3, 4]
        ax_b       = [47, 3, 47]
        axes       = [0, 2]
        b          = array([[[0.12306367, 0.3721741 , 0.5628895 , ..., 0.3801674 ,
         0.74808821, 0.13740255],
        [0.12073513, 0...58957  , 0.08708803],
        [0.80827012, 0.1201999 , 0.55677126, ..., 0.30253992,
         0.50004582, 0.80420667]]])
        is_complex = False
        mode       = 'same'
        monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f2a6e1c4bd0>
        self       = <scipy.signal.tests.test_signaltools.TestOAConvolve object at 0x7f2a6e196350>
        shape_a_0  = 4
        shape_a_1  = 4
        shape_a_extra = 3
        shape_b_0  = 47
        shape_b_1  = 47
        shape_b_extra = 3
scipy/signal/_signaltools.py:671: in fftconvolve
    ret = _freq_domain_conv(in1, in2, axes, shape, calc_fast_len=True)
        axes       = [0, 2]
        in1        = array([[[0.88156736, 0.86223916, 0.16972094, 0.89005165],
        [0.15585618, 0.624305  , 0.70602407, 0.7034736 ],
  ...,
        [0.77882918, 0.50795249, 0.24284804, 0.85650584],
        [0.35205817, 0.42588504, 0.05523513, 0.65621471]]])
        in2        = array([[[0.12306367, 0.3721741 , 0.5628895 , ..., 0.3801674 ,
         0.74808821, 0.13740255],
        [0.12073513, 0...58957  , 0.08708803],
        [0.80827012, 0.1201999 , 0.55677126, ..., 0.30253992,
         0.50004582, 0.80420667]]])
        mode       = 'same'
        s1         = (4, 3, 4)
        s2         = (47, 3, 47)
        shape      = [50, 3, 50]
scipy/signal/_signaltools.py:508: in _freq_domain_conv
    sp1 = fft(in1, fshape, axes=axes)
        axes       = [0, 2]
        calc_fast_len = True
        complex_result = False
        fft        = <uarray multimethod 'rfftn'>
        fshape     = [50, 50]
        ifft       = <uarray multimethod 'irfftn'>
        in1        = array([[[0.88156736, 0.86223916, 0.16972094, 0.89005165],
        [0.15585618, 0.624305  , 0.70602407, 0.7034736 ],
  ...,
        [0.77882918, 0.50795249, 0.24284804, 0.85650584],
        [0.35205817, 0.42588504, 0.05523513, 0.65621471]]])
        in2        = array([[[0.12306367, 0.3721741 , 0.5628895 , ..., 0.3801674 ,
         0.74808821, 0.13740255],
        [0.12073513, 0...58957  , 0.08708803],
        [0.80827012, 0.1201999 , 0.55677126, ..., 0.30253992,
         0.50004582, 0.80420667]]])
        shape      = [50, 3, 50]
scipy/fft/_backend.py:28: in __ua_function__
    return fn(*args, **kwargs)
        args       = (array([[[0.88156736, 0.86223916, 0.16972094, 0.89005165],
        [0.15585618, 0.624305  , 0.70602407, 0.7034736 ],
 ...0.77882918, 0.50795249, 0.24284804, 0.85650584],
        [0.35205817, 0.42588504, 0.05523513, 0.65621471]]]), [50, 50])
        fn         = <function rfftn at 0x7f2a7238a020>
        kwargs     = {'axes': [0, 2]}
        method     = <uarray multimethod 'rfftn'>
scipy/fft/_basic_backend.py:119: in rfftn
    return _execute_nD('rfftn', _pocketfft.rfftn, x, s=s, axes=axes, norm=norm,
        axes       = [0, 2]
        norm       = None
        overwrite_x = False
        plan       = None
        s          = [50, 50]
        workers    = None
        x          = array([[[0.88156736, 0.86223916, 0.16972094, 0.89005165],
        [0.15585618, 0.624305  , 0.70602407, 0.7034736 ],
  ...,
        [0.77882918, 0.50795249, 0.24284804, 0.85650584],
        [0.35205817, 0.42588504, 0.05523513, 0.65621471]]])
scipy/fft/_basic_backend.py:45: in _execute_nD
    return pocketfft_func(x, s=s, axes=axes, norm=norm,
        axes       = [0, 2]
        func_str   = 'rfftn'
        norm       = None
        overwrite_x = False
        plan       = None
        pocketfft_func = functools.partial(<function r2cn at 0x7f2a72388f40>, True)
        s          = [50, 50]
        workers    = None
        x          = array([[[0.88156736, 0.86223916, 0.16972094, 0.89005165],
        [0.15585618, 0.624305  , 0.70602407, 0.7034736 ],
  ...,
        [0.77882918, 0.50795249, 0.24284804, 0.85650584],
        [0.35205817, 0.42588504, 0.05523513, 0.65621471]]])
        xp         = <module 'scipy._lib.array_api_compat.array_api_compat.numpy' from '/home/treddy/github_projects/scipy/build-install/lib/python3.11/site-packages/scipy/_lib/array_api_compat/array_api_compat/numpy/__init__.py'>
scipy/fft/_pocketfft/basic.py:179: in r2cn
    return pfft.r2c(tmp, axes, forward, norm, None, workers)
        _          = True
        axes       = [0, 2]
        forward    = True
        norm       = 0
        overwrite_x = False
        plan       = None
        s          = [50, 50]
        shape      = (50, 50)
        tmp        = array([[[0.88156736, 0.86223916, 0.16972094, ..., 0.        ,
         0.        , 0.        ],
        [0.15585618, 0...       , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ]]])
        workers    = 1
        x          = array([[[0.88156736, 0.86223916, 0.16972094, 0.89005165],
        [0.15585618, 0.624305  , 0.70602407, 0.7034736 ],
  ...,
        [0.77882918, 0.50795249, 0.24284804, 0.85650584],
        [0.35205817, 0.42588504, 0.05523513, 0.65621471]]])
/home/treddy/python_venvs/py_311_scipy_dev/lib/python3.11/site-packages/numpy/core/multiarray.py:15: in __getattr__
    _raise_warning(attr_name, "multiarray")
        _raise_warning = <function _raise_warning at 0x7f2a740d6480>
        attr_name  = '_ARRAY_API'
        multiarray = <module 'numpy._core.multiarray' from '/home/treddy/python_venvs/py_311_scipy_dev/lib/python3.11/site-packages/numpy/_core/multiarray.py'>
        ret        = <capsule object NULL at 0x7f2a766938a0>
/home/treddy/python_venvs/py_311_scipy_dev/lib/python3.11/site-packages/numpy/core/_utils.py:10: in _raise_warning
    warnings.warn(
E   DeprecationWarning: numpy.core.multiarray is deprecated and has been renamed to numpy._core.multiarray. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.multiarray._ARRAY_API.
        attr       = '_ARRAY_API'
        new_module = 'numpy._core.multiarray'
        old_module = 'numpy.core.multiarray'
        submodule  = 'multiarray'



### Is this a regression? Put the last known working version here if it is.

Not a regression

Ah, looks fixed on latest master (I checked locally) after @ngoldbaum pointed out the fix, but not out in a stable release yet. Well, I do have 1 failure left when I use latest master, but that may be from another prebuild dependency I think:

_____________________________________________________________________________________________________ TestPlotting.test_delaunay _____________________________________________________________________________________________________
[gw30] linux -- Python 3.11.2 /home/treddy/python_venvs/py_311_scipy_dev/bin/python
scipy/spatial/tests/test__plotutils.py:28: in test_delaunay
    r = delaunay_plot_2d(obj, ax=fig.gca())
        fig        = <Figure size 640x480 with 1 Axes>
        obj        = <scipy.spatial._qhull.Delaunay object at 0x7f4069c8e090>
        s_before   = array([[3, 1, 0],
       [2, 3, 0]], dtype=int32)
        self       = <scipy.spatial.tests.test__plotutils.TestPlotting object at 0x7f406fc45690>
        sup        = <numpy.testing._private.utils.suppress_warnings object at 0x7f406ae7ded0>
<decorator-gen-2>:2: in delaunay_plot_2d
    ???
        ax         = <Axes: >
        tri        = <scipy.spatial._qhull.Delaunay object at 0x7f4069c8e090>
scipy/spatial/_plotutils.py:20: in _held_figure
    return func(obj, ax=ax, **kw)
        ax         = <Axes: >
        func       = <function delaunay_plot_2d at 0x7f4084c245e0>
        kw         = {}
        obj        = <scipy.spatial._qhull.Delaunay object at 0x7f4069c8e090>
        plt        = <module 'matplotlib.pyplot' from '/home/treddy/python_venvs/py_311_scipy_dev/lib/python3.11/site-packages/matplotlib/pyplot.py'>
        was_held   = True
scipy/spatial/_plotutils.py:86: in delaunay_plot_2d
    ax.triplot(x, y, tri.simplices.copy())
        ax         = <Axes: >
        tri        = <scipy.spatial._qhull.Delaunay object at 0x7f4069c8e090>
        x          = array([0., 0., 1., 1.])
        y          = array([0., 1., 0., 1.])
/home/treddy/python_venvs/py_311_scipy_dev/lib/python3.11/site-packages/matplotlib/tri/_triplot.py:41: in triplot
    x, y, edges = (tri.x, tri.y, tri.edges)
        args       = []
        ax         = <Axes: >
        kwargs     = {}
        matplotlib = <module 'matplotlib' from '/home/treddy/python_venvs/py_311_scipy_dev/lib/python3.11/site-packages/matplotlib/__init__.py'>
        tri        = <matplotlib.tri._triangulation.Triangulation object at 0x7f4069c17710>
/home/treddy/python_venvs/py_311_scipy_dev/lib/python3.11/site-packages/matplotlib/tri/_triangulation.py:114: in edges
    self._edges = self.get_cpp_triangulation().get_edges()
        self       = <matplotlib.tri._triangulation.Triangulation object at 0x7f4069c17710>
/home/treddy/python_venvs/py_311_scipy_dev/lib/python3.11/site-packages/matplotlib/tri/_triangulation.py:124: in get_cpp_triangulation
    self._cpp_triangulation = _tri.Triangulation(
        _tri       = <module 'matplotlib._tri' from '/home/treddy/python_venvs/py_311_scipy_dev/lib/python3.11/site-packages/matplotlib/_tri.cpython-311-x86_64-linux-gnu.so'>
        self       = <matplotlib.tri._triangulation.Triangulation object at 0x7f4069c17710>
/home/treddy/python_venvs/py_311_scipy_dev/lib/python3.11/site-packages/numpy/core/multiarray.py:15: in __getattr__
    _raise_warning(attr_name, "multiarray")
        _raise_warning = <function _raise_warning at 0x7f408636db20>
        attr_name  = '_ARRAY_API'
        multiarray = <module 'numpy._core.multiarray' from '/home/treddy/python_venvs/py_311_scipy_dev/lib/python3.11/site-packages/numpy/_core/multiarray.py'>
        ret        = <capsule object NULL at 0x7f408c795e60>
/home/treddy/python_venvs/py_311_scipy_dev/lib/python3.11/site-packages/numpy/core/_utils.py:10: in _raise_warning
    warnings.warn(
E   DeprecationWarning: numpy.core.multiarray is deprecated and has been renamed to numpy._core.multiarray. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.multiarray._ARRAY_API.
        attr       = '_ARRAY_API'
        new_module = 'numpy._core.multiarray'
        old_module = 'numpy.core.multiarray'
        submodule  = 'multiarray'

pybind / pybind11