search

cubed-dev / cubed

Bounded-memory serverless distributed N-dimensional array processing
https://cubed-dev.github.io/cubed/
Apache License 2.0
122 stars 14 forks source link

Implement NumPy's `__array_function__` protocol for array methods that are not in the Array API Standard #475

Open tomwhite opened 5 months ago

tomwhite commented 5 months ago

This is to support nanmean, nansum (see #153), and pad (#461).

The previous attempt at this in #468 was reverted in #469, since it was causing problems when used with Xarray (https://github.com/pydata/xarray/issues/8834#issuecomment-2124914727).

When Xarray is working we can re-introduce the changes here.

tomwhite commented 1 month ago

The Xarray change that enables this has been merged (https://github.com/pydata/xarray/pull/9530).

I've re-created the branch from #468 here: https://github.com/cubed-dev/cubed/tree/np-array-function. However, when I run the cubed-xarray tests I get some failures:

FAILED cubed_xarray/tests/test_wrapping.py::test_to_zarr[single-threaded] - TypeError: no implementation found for 'numpy.isclose' on types that implement __array_function__: [<class 'cubed.array_api.a...
FAILED cubed_xarray/tests/test_wrapping.py::test_to_zarr[processes] - TypeError: no implementation found for 'numpy.isclose' on types that implement __array_function__: [<class 'cubed.array_api.a...
FAILED cubed_xarray/tests/test_xarray.py::TestVariable::test_binary_op_bitshift - TypeError: no implementation found for 'numpy.full_like' on types that implement __array_function__: [<class 'cubed.array_api...
FAILED cubed_xarray/tests/test_xarray.py::TestVariable::test_reduce - TypeError: no implementation found for 'numpy.isclose' on types that implement __array_function__: [<class 'cubed.array_api.a...
FAILED cubed_xarray/tests/test_xarray.py::TestDataArrayAndDataset::test_lazy_array - TypeError: no implementation found for 'numpy.isclose' on types that implement __array_function__: [<class 'cubed.array_api.a...
FAILED cubed_xarray/tests/test_xarray.py::TestDataArrayAndDataset::test_dataarray_pickle - TypeError: no implementation found for 'numpy.full_like' on types that implement __array_function__: [<class 'cubed.array_api...
FAILED cubed_xarray/tests/test_xarray.py::TestDataArrayAndDataset::test_dataset_pickle - AssertionError: Left and right Dataset objects are not identical
FAILED cubed_xarray/tests/test_xarray.py::test_unify_chunks - AssertionError: Left and right Dataset objects are not identical
================================ 8 failed, 41 passed, 2 skipped, 15 xfailed, 3 warnings in 3.47s =================================

There is no isclose function in the Array API (https://github.com/data-apis/array-api/issues/170), so we may need to build one (in Xarray?).

cc @keewis @TomNicholas