What's missing from xarray?

[TomNicholas]

Hi! I just saw this project, and I'm wondering why exactly you felt xarray wasn't suitable for your use case?

In the readme xarray is mentioned, but it's not totally clear to me what xarray doesn't offer that you need?

We even have an in-progress project in xarray to split out a lightweight version of the xr.Variable object as an xr.NamedArray, which seems to be exactly what you're trying to build here.

I would love to see more collaboration between the (awesome) astropy stack and the xarray community, so please let us know if we can help here!

[byrdie]

Hi @TomNicholas, and thanks for your interest in my project! I contemplate daily whether I should've just contributed to or extended xarray, but I'm afraid I've gone too far to change course at the moment for my other projects.

Apologies for the sparse documentation right now, I've been meaning to create a page which describes why I decided to go in a slightly different direction than xarray.

In short, the reasons I can remember for starting a different project (in no particular order) are as follows. Some of these issues may have been fixed by features added to xarray after I started development on this project, so please let me know if anything is incorrect or out of date.

• First-class Astropy units support. Arrays in this project are all duck versions of astropy.Quantity, with a units attribute and other Quantity-specific features. I'm familiar with pint-xarray, I'm sure a similar thing could be done with Quantities (see astropy/astropy#14454).
• Abstract arrays. Quite controversial to use inheritance, but I really wanted different array types like a LinearSpace array analogous to np.linspace that would make it easier for certain optimizations to be carried out, for example when interpolating.
• Vector and matrix support. pydata/xarray#5775, mentions this, but I needed types like Cartesian3DVector and Cartesian3dMatrix to make it easier to do linear algebra operations and I already had very strong opinions on how I wanted it implemented. I probably could've extended xarray for this feature.
• Uncertainty propagation. I needed uncertainties to propagate through all array operations seamlessly, in a similar manner to astropy.uncertainties, and I needed to set up the multiple dispatch in a particular way to make it work properly.
• Generic types. I haven't totally got this working, but I wanted all arrays to be generic so that mypy would work properly for array elements.
• No positional indexing allowed. I decided that I thought positional indexing was harmful, so named-arrays only allows indexing by name.

I can certainly imagine that xarray will have all these features and more in the near future, and I would be interested in contributing what I can to xarray as soon as I'm done with my current project.

Looking forward to discussing this further, thanks again for raising this issue!

[byrdie]

So after perusing the xarray source this morning, I realize that a few of my points are obviously wrong.

• Abstract arrays. xarray has an AbstractArray class, much the same as my implementation, I don't know why I was under the opposite impression, apologies!
• Generic types. I see that xr.NamedArray is generic, so this comment is out-of-date.

[byrdie]

And regarding further collaboration with the Astropy stack, I think the planned Quantity 2.0 project, astropy/astropy-APEs#91, will make this very easy, since xr.NamedArray will be able to just wrap this new Quantity 2.0 object and probably most of the array API would just work.

[TomNicholas]

Thanks for the detailed response!

xarray has an AbstractArray class, much the same as my implementation, I don't know why I was under the opposite impression, apologies!

You can also subclass xarray objects, though is it rarely done and probably has some sharp edges.

No positional indexing allowed. I decided that I thought positional indexing was harmful, so named-arrays only allows indexing by name.

Yeah we do still allow it, but it's a rarely-used feature. I personally never do it. I agree with you that it might be better if xarray just forbid it entirely, but I also think it's a feature that can be ignored if you don't like it.

Vector and matrix support.

Yeah I would extend xarray if you want to add this kind of thing. Xarray is not trying to represent vector calculus - anyone who wants to do that should probably build their own project on top of xarray.

Uncertainty propagation.

This is something that hopefully could be done at the level of the wrapped array, and it would be great if someone thought seriously about how to do this with xarray.

I think the planned Quantity 2.0 project, https://github.com/astropy/astropy-APEs/pull/91, will make this very easy, since xr.NamedArray will be able to just wrap this new Quantity 2.0 object and probably most of the array API would just work.

Thanks for linking to this!