RFC:DOC: Better documentation of functions that support alternative backends

[j-bowhay]

Currently, we list functions that support the array API on https://scipy.github.io/devdocs/dev/api-dev/array_api.html#currently-supported-functionality . Initially, while support was very limited, this was a good solution. However, I believe we are now outgrowing it as more functions gain support. I would like to propose we discuss a long term solution to how we document array API support. I think this is necessary for the following reasons:

• Visibility. The current documentation is not very visible and requires you to go back and forth between the API docs, which is annoying as a user. Whilst this is still an experimental feature, I believe we should start to draw more attention to array API support so that we can gain more feedback and users before flicking the switch.
• Partial support: Currently it is not clear which functions support all array backends vs CPU only. Also, some functions don't work with JAX etc
• Delegation: I think it is important the documentation is clear about when we delegate to another library ie as we do with special functions. Whilst normally the user should not have to care about implementation details, I would argue that sometimes scientific computing is different. Different implementations may have different stability and performance characteristics which users care about. For example, with special some implementations have subpar accuracy for certain functions/inputs so it is important the user knows where their function call is "going".
• Maintainability: The current list is a bit of a chore to maintain being in sentence format as if every array API PR added one function to this list there would be a lot of merge conflicts. This means in practice we wait and add them in bulk before a release. This is easy to forget (ie last release)!

I propose we maintain some sort of central record that allows us to record functions/modules that support the array API, which array types they support, and where/which libraries they delegate to. For this central record, we could then automatically create a table of support that supersedes our current documentation. We could also automatically inject into the API documentation pages to improve the visibility of functions that support the array API.

cc @rgommers

[j-bowhay]

Also in the case of datasets etc which will always return NumPy arrays it should be made clear that it is the user's job to then do xp.asarray on the output

[rgommers]

Thanks for this @j-bowhay! I agree that it's time to make this more visible in the docs, and also that we need a structured and maintainable way of recording this metadata and translating it into html docs.

One important point that we discussed in the last community meeting is that while we want both a single-page (or per-submodule) overview as well as info in the individual function documentation, we probably do not want to dynamically update all the docstrings at import time, to avoid a large import time hit (this would need checking, but feels expensive).

I imagine we'd want some static metadata file (a la scipy/special/functions.json) where we capture the state of each function. A function gets an entry once it supports more than only numpy, and we'd need to record the "works with" for various libraries and devices, resulting in this kind of info (x for yes, empty for no):

Array library	CPU	GPU	JIT
NumPy	x	n/a
PyTorch
JAX
CuPy	n/a
any array API compliant

The JIT column could be added later, not quite sure - it does matter, but we don't have test coverage. For NumPy the JIT is Numba, for PyTorch it's "works with torch.compile", for JAX it's "works with jax.jit", etc.

[rgommers]

Delegation: I think it is important the documentation is clear about when we delegate to another library

This is the one bit I don't agree with. It's important to maintain the separation between what's public API/semantics and what are internal implementation details. For CuPy/PyTorch/JAX we will always delegate back to those libraries for execution; we are just calling different functions if we have wrappers like in special. Also within SciPy we often have multiple code paths for the same function, depending on input array dtype or something like that. This is not something we should be documenting structurally and exposing to end users. Of course for individual functions there may be reasons to do so, but it can't really be the right thing to do in general.

with special some implementations have subpar accuracy for certain functions/inputs

This is also always going to be the case when there are multiple code paths, it will never be identical. When code passes the tests though with normal tolerances, we have no reason to expose details on this to end users. We should only document things that are far outside of the typically expected accuracy, or known bugs/issues.

[j-bowhay]

One additional idea I had (which might be over-engineering things) is that the metadata file should be the single source of truth and should control which backends a function's tests are run against. Otherwise, things could quickly get out of sync, leading to gaps in test coverage.