Open eric-czech opened 4 years ago
cf. this thread on dispatch in Xarray: https://github.com/pydata/xarray/issues/1938
Some interesting discussion also happening at https://github.com/pydata/xarray/issues/3213#issuecomment-615772303 with regards to scipy.sparse
and pydata/sparse
, which may be "backends" to consider as well.
Some more notes/questions:
da.map_blocks(backend_module.asarray)
after reads, but some readers may be much more efficient if they aren't loading chunks as some default duck array type (probably numpy) and then undergoing conversionShould our genetics methods assume that the same target array backend can be used for all n-ary numpy functions?
SparseBackend
and that the method produces dense results, attempting to stack them will fail:import unumpy as unp
import uarray as ua
import unumpy.sparse_backend as SparseBackend
with ua.set_backend(SparseBackend):
unp.stack([np.array([1]), np.ones([1])])
# ValueError: All arrays must be instances of SparseArray.
getattr(dask.array if dask_installed else numpy, numpy_function)(*args, **kwargs)
for handling chunked vs. unchunked dispatching
Two immediately necessary uses for this are:
The second is far more complicated and a separate framework may not be necessary for the first, but it would be great to support both the same way.
On array dispatch, I think dispatching based on argument types is not enough. We will likely have many functions that take multiple array args and if they are a mix of dask/numpy/sparse arrays, a better solution to supporting this is likely to have the user declare what backend API should be preferred and then special-case coercion where necessary.
At a minimum, I think we should keep CuPy, Dask, and Numpy backends in mind since we already know how different implementations of genetic algorithms are going to be based on Alistair's skallel v2 prototype. Each backend will definitely need to use API-specific functionality but a lot of operations will be dispatchable purely through the numpy API too. A good question to answer would be whether or not literally using numpy is better for the latter or if unumpy will make more sense. The backend dispatching model in unumpy seems like a good fit but I don't know if aligning to this long-term is worth the extra dependencies. I think it will depend on how much non API-specific code we actually need.