Provide coordinates for both `time` and `frame`

[niksirbi]

Is your feature request related to a problem? Please describe. Currently in our xarray.Dataset we only have one time axis, the units of which may vary depending on whether the user has provided an fps value or not. This is a bit confusing, the coords of that axis may be in seconds or in frames. We keep track of the units in Dataset.attrs, but it may still cause confusion.

Describe the solution you'd like We could have both coordinates systems -frame and time (in seconds) associated with the same axis/dimension. The user should be able to do ds.sel(frame=10) as well as ds.sel(time=200). AFAIK this is supported by xarray, see example implementation in xarray-video.

Describe alternatives you've considered Alternatively we could alway require fps and only have the time coords in seconds. But even if we do that, it's still valuable to have both time and frame coords defined, as users might need one or the other, depending on the occasion.

[niksirbi]

What do y'all think @neuroinformatics-unit/behaviour @b-peri ?

[adamltyson]

Both sounds like a good idea to me.

[b-peri]

I agree - having both would be super useful!

[niksirbi]

After discussing at the team meeting today, we settled on the following course of action:

• frame axis should always be created when loading the data
• fps is optional, but if the user provides it we also create a time axis in seconds.
• We also need a way for people to provide the fps later, if they didn't do so during file loading.
• If the user requests an operation that requires the time axis, but this doesn't exist yet, we should prompt them to supply the fps first (e.g. in the error message).

[talmo]

An option to provide explicit vector of timestamps for the time axis would cover this common scenario.

Great point, also useful for synchronising with other modalities. We should definitely provide a method for people to pass a vector of timestamps instead of fps.

[niksirbi]

xarray supports dimensions with multiuple coordinate axes, but we'd have to carefully review the relevant terminilogy of dimension- vs non-dimension coordinates and indexed-vs non-indexed coordinates. Ideally we want both time and frames to be indexed coordinates, i.e. ones that permit slicing.