Open fujiisoup opened 6 years ago
Copying @horta's doc:
A data array a
has d
dimensions, ordered from 0
to d
. It contains an array of dimensionality d
. The first dimension of that array is associated with the first dimension of the data array, and so forth. That array is returned by the data array attribute values
. A named data array is a data array with the name
attribute of string value.
Each data array dimension has an unique name
attribute of string type and can be accessed via data array dims
attribute of tuple type. The name of the dimension i
is a.dims[i]
.
A data array can have zero or more coordinates, represented by a dict-like coords
attribute. A coordinate is a named data array, referred also as coordinate data array. Coordinate data arrays have unique names among other coordinate data arrays. A coordinate data array of name x
can be retrieved by a.coords[x]
.
A coordinate can have zero or more dimensions associated with. A dimension data array is a unidimensional coordinate data array associated with one, and only one, dimension having the same name as the coordinate data array itself. A dimension data array has always one, and only one, coordinate. That coordinate has again a dimension data array associated with.
As an example, let a.coords[name]
be a dimension data array of a
, associated with dimension i
. This means, among other things, that
a.shape[i] == a.coords[name].size
a.coords[name].name == name
len(a.coords[name].dims) == 1
a.coords[name].dims[0] == name
Now lets consider a practical example:
>>> import numpy as np
>>> import xarray as xr
>>>
>>> a = xr.DataArray(np.arange(6).reshape((3, 2)), dims=["x", "y"], coords={"x": list("abc")})
>>> a
<xarray.DataArray (x: 3, y: 2)>
array([[0, 1],
[2, 3],
[4, 5]])
Coordinates:
* x (x) <U1 'a' 'b' 'c'
Dimensions without coordinates: y
Data array a
has two dimensions: "x"
and "y"
. It has a single coordinate "x"
, with its associated dimension data array a.coords["x"]
. The dimension data array definition implies in the following recursion:
>>> a.coords["x"]
<xarray.DataArray 'x' (x: 3)>
array(['a', 'b', 'c'], dtype='<U1')
Coordinates:
* x (x) <U1 'a' 'b' 'c'
>>> a.coords["x"].coords["x"]
<xarray.DataArray 'x' (x: 3)>
array(['a', 'b', 'c'], dtype='<U1')
Coordinates:
* x (x) <U1 'a' 'b' 'c'
Indexing is a operation that when applied to a data array will produce a new data array according the rules explained here.
The most general form of indexing is the one that involves data arrays i0, i1, ...
in the operation a.sel(d0=i0, d1=i1, ...)
, for which d0, d1, ...
are an exhaustive list of dimension names of a
. i0
will select the indices from dimension d0
that match with its indices. And the same happens with each other dimension, independently. The matching works like an SQL JOIN:
i0
are the values of d0
after flattening the associated array.The following code snippet shows an indexing example of a data array a
with two indexers i0
and i1
.
>>> a = xr.DataArray(np.arange(6).reshape((3, 2)), dims=["x", "y"], coords={"x": list("abc"), "y": [0, 1]})
>>> i0 = xr.DataArray(['a', 'c'], dims=["x"])
>>> i0
<xarray.DataArray (x: 2)>
array(['a', 'c'], dtype='<U1')
Dimensions without coordinates: x
>>> i1 = xr.DataArray([0, 1], dims=["y"])
>>> i1
<xarray.DataArray (y: 2)>
array([0, 1])
Dimensions without coordinates: y
>>> a.sel(x=i0, y=i1)
<xarray.DataArray (x: 2, y: 2)>
array([[0, 1],
[4, 5]])
Coordinates:
* x (x) <U1 'a' 'c'
* y (y) int64 0 1
As hinted above, xarray allows the use of multidimensional data array indexers for greater flexibility. Lets look at another exampe:
>>> a = xr.DataArray([0, 1, 2], dims=["x"], coords={"x": list("abc")})
>>> a
<xarray.DataArray (x: 3)>
array([0, 1, 2])
Coordinates:
* x (x) <U1 'a' 'b' 'c'
>>> i0 = xr.DataArray([['a', 'c']], dims=["x0", "x1"])
>>> i0
<xarray.DataArray (x0: 1, x1: 2)>
array([['a', 'c']], dtype='<U1')
Dimensions without coordinates: x0, x1
>>> a.sel(x=i0)
<xarray.DataArray (x0: 1, x1: 2)>
array([[0, 2]])
Coordinates:
x (x0, x1) object 'a' 'c'
Dimensions without coordinates: x0, x1
The resulting data array have the same dimensions as the indexer, not as the original data array. Notice also that the resulting data array has no dimension data array as opposed to the previous example. Instead, it has a bi-dimensional coordinate data array:
>>> a.sel(x=i0).coords["x"]
<xarray.DataArray 'x' (x0: 1, x1: 2)>
array([['a', 'c']], dtype=object)
Coordinates:
x (x0, x1) object 'a' 'c'
Dimensions without coordinates: x0, x1
Thanks guys! Just to make sure, this is a work in progress. i realise that I made some wrong assumptions, and there are more to add into it.
OK, maybe I was premature to copy it here :). But yes, I'm excited about clarifying these definitions in our docs!
I have updated mainly the Indexing and selection data section. I'm proposing an indexing notation using []
operator vs ()
function call to differentiate between dimension lookup. But more importantly, I'm working out a precise definition of data array indexing in section Formal indexing definition.
A data array a
has D
dimensions, ordered from 0
to D
. It contains an array of dimensionality D
. The first dimension of that array is associated with the first dimension of the data array, and so forth. That array is returned by the data array attribute values
. A named data array is a data array with the name
attribute of string value:
>>> import xarray as xr
>>>
>>> a = xr.DataArray([[0, 1], [2, 3], [4, 5]])
>>> a.name = "My name"
>>> a
<xarray.DataArray 'My name' (dim_0: 3, dim_1: 2)>
array([[0, 1],
[2, 3],
[4, 5]])
Dimensions without coordinates: dim_0, dim_1
Each data array dimension has an unique name
attribute of string type and can be accessed via data array dims
attribute of tuple type. The name of the dimension i
is a.dims[i]
:
>>> a.dims[0]
'dim_0'
A data array can have zero or more coordinates, represented by a dict-like coords
attribute. A coordinate is a named data array, referred also as coordinate data array. Coordinate data arrays have unique names among other coordinate data arrays. A coordinate data array of name x
can be retrieved by a.coords[x]
.
A coordinate can have zero or more dimensions associated with. A dimension data array is a unidimensional coordinate data array associated with one, and only one, dimension having the same name as the coordinate data array itself. A dimension data array has always one, and only one, coordinate. That coordinate has again a dimension data array associated with:
>>> import numpy as np
>>>
>>> a = xr.DataArray(np.arange(6).reshape((3, 2)), dims=["x", "y"], coords={"x": list("abc")})
>>> a
<xarray.DataArray (x: 3, y: 2)>
array([[0, 1],
[2, 3],
[4, 5]])
Coordinates:
* x (x) <U1 'a' 'b' 'c'
Dimensions without coordinates: y
The above data array a
has two dimensions: "x"
and "y"
. It has a single coordinate "x"
, with its associated dimension data array a.coords["x"]
. The dimension data array definition implies in the following recursion:
>>> a.coords["x"]
<xarray.DataArray 'x' (x: 3)>
array(['a', 'b', 'c'], dtype='<U1')
Coordinates:
* x (x) <U1 'a' 'b' 'c'
>>> a.coords["x"].coords["x"]
<xarray.DataArray 'x' (x: 3)>
array(['a', 'b', 'c'], dtype='<U1')
Coordinates:
* x (x) <U1 'a' 'b' 'c'
Coordinate data arrays are meant to provide labels to array positions, allowing for convenient access to array elements:
>>> a.loc["b", :]
<xarray.DataArray (y: 2)>
array([2, 3])
Coordinates:
x <U1 'b'
Dimensions without coordinates: y
Note that there is no asterisk symbol for coordinate "x"
of the above resulting data array, as the coordinate is not associated with any dimension. In other words, the coordinate data array a.loc["b", :].coords["x"]
is not a dimension data array.
There are four different but equally powerful ways of selecting data from a data array. They differ only on the type of dimension and index lookups: position-based lookup and label-based lookup:
| Dimension lookup | Index lookup | Data array |
|------------------|----------------|--------------------|
| Position-based | Position-based | a[:, 0] |
| Position-based | Label-based | a.loc[:, "UK"] |
| Label-based | Position-based | a(country=0) |
| Label-based | Label-based | a.loc(country="UK")|
A dimension position-based lookup is determined by the used position in the index operator: a[first_dim, second_dim, ...]
and a.loc[first_dim, second_dim, ...]
. An index position-based lookup is determined by the provided integers or slices: a[0, [3, 9], :, ...]
and a.loc(country=0, time=[3, 9], space=slice(None))
.
A dimension label-based lookup is determined by the provided dimension name: a(country=0)
and a.loc(country="UK")
. An index label-based loookup is determined by the provided index labels or slices [1]: a.loc[:, "UK"]
and a.loc(countr="UK")
.
[1] An index label is any Numpy data type object.
Consider the following data array:
>>> a = xr.DataArray(np.arange(6).reshape((3, 2)), dims=["year", "country"], coords={"year": [1990, 1994, 1998], "country": ["UK", "US"]})
>>> a
<xarray.DataArray (year: 3, country: 2)>
array([[0, 1],
[2, 3],
[4, 5]])
Coordinates:
* year (year) int64 1990 1994 1998
* country (country) <U2 'UK' 'US'
The expressions a[:, 0]
, a.loc[:, "UK"]
, a(country=0)
, and a.loc(country="UK")
will all produce the same result:
>>> a.loc[:, "UK"]
<xarray.DataArray (year: 3)>
array([0, 2, 4])
Coordinates:
* year (year) int64 1990 1994 1998
country <U2 'UK'
Let A
be the dimensionality of the a
, the data array being indexed. Let b
be the resulting data array. The Python operations for indexing is formally translated into an A
-tuple:
(i_1, i_2, ..., i_A)
for which i_j
is a named data array whose values are index labels. This data array, as usual, can have 0
, 1
, or more dimensions. Its construction is described later in this section.
Let
(r_1, r_2, ..., r_A)
be the tuple representing the indices of a
. Precisely, temporarily create dimension data arrays with labels from 0
to the dimension size for those dimensions without an associated dimension data array. Therefore, r_j
is a dimension data array for dimension j
. Also, it is required that i_j
values are a subset of r_j
values.
For each j
, define the lists I_j = [(i_j0, 0), (i_j1, 1), ...]
and R_j = [(r_j0, 0), (r_j1, 1), ...]
with pairs of data array values and positions. Perform a SQL JOIN as follows:
I_j
and the values of R_j
.Consider i_0
and r_0
defined as follows:
>>> i_0
<xarray.DataArray (apples: 2, oranges: 1)>
array([['a'],
['c']], dtype='<U1')
Dimensions without coordinates: apples, oranges
>>> r_0
<xarray.DataArray (dim_0: 3)>
array(['a', 'b', 'c'], dtype='<U1')
Coordinates:
* dim_0 (dim_0) <U1 'a' 'b' 'c'
Performing operations 1 and 2 will result in the list IR_j = [(('a', 0), ('a', 0)), (('c', 1), ('c', 3))]
.
The positions IR_j[0][1][1]
, IR_j[1][1][1]
, and so forth are used to access the values of a
and assign them to the positions IR_j[0][0][1]
, IR_j[1][0][1]
, and so forth of b
. Precisely, for each (t_0, t_1, ..., t_A)
in itertools.product(IR_0, IR_1, ..., IR_A)
, assign
b[reshape(t_0[0][1], i_0.shape), ..., reshape(t_A[0][1], i_A.shape)] = a[t_0[1][1], ..., t_A[1][1]]
Thanks, @horta
Your description looks much more exact than that we already have in our doc. It would help readers when they want to know the data structure of xarray more deeply.
I am a little wondering how we could balance the exactness and simplicity of the description. I don't think it's a good idea to simply replace this section by your more strict definition, as it would be hard for beginners to grasp the outline of this section.
I think it would be nice if we can have both
Any idea about how our docs should looks like?
I will first try to have both together. I'm well aware that learning by examples (that is true for me at least and apparently to most of people: tldr library), so at first I will try to combine all in one page:
I prefer starting with 2 and 3 for me to actually understand xarray...
@horta, what's the status on this? @max-sixty and I were discussing a glossary over in https://github.com/pydata/xarray/issues/3176. I think this is an important contribution and am happy to help if needed.
In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity
If this issue remains relevant, please comment here or remove the stale
label; otherwise it will be marked as closed automatically
We have a version here now: https://xarray.pydata.org/en/stable/user-guide/terminology.html
Actualy we should update the indexing.rst
page with the nice text here.
It appears there is an incorrect statement in the 'indexing' section of the terminology page.
By construction,
len(arr.dims) == len(arr.indexes)
But this simple code does not reproduce this result
da = xr.DataArray(np.random.randn(2, 3), dims=("x", "y"), coords={"x": [10, 20]})
len(da.dims) == len(da.indexes)
>False
Because the dimension y does not have any coordinates associated with it. Maybe I am missing something, but it seems a clarification should be added that this is only true for dimensions with coordinates assigned to them.
We have a bunch of terms to describe the xarray structure, such as dimension, coordinate, dimension coordinate, etc.. Although it has been discussed in #1295 and we tried to use the consistent terminology in our docs, it looks still not easy for users to understand our functionalities.
In #2399, @horta wrote a list of definitions (https://drive.google.com/file/d/1uJ_U6nedkNe916SMViuVKlkGwPX-mGK7/view?usp=sharing). I think it would be nice to have something like this in our docs.
Any thought?