Representing & checking Dataset schemas

[max-sixty]

What would be the best way to canonically describe a dataset, which could be read by both humans and machines?

For example, frequently in our code we have docstrings which look something like:

def get_returns(security_ids):
    """
    Retuns mega-dimensional dataset which gives recent returns for a set of
        securities by:
    - Date
    - Return (raw / economic / smoothed / etc)
    - Scaling (constant / risk_scaled)
    - Span
    - Hedged vs Unhedged

    Dataset keys are security ids. All dimensions have coords.
    """

This helps when attempting to understand what code is doing while only reading it. But this isn't consistent between docstrings and can't be read or checked by a machine. Has anyone solved this problem / have any suggestions for resources out there?

Tangentially related to https://github.com/python/typing/issues/513 (but our issues are less about the type, dimension sizes, and more about the arrays within a dataset, their dimensions, and their names)

[shoyer]

I think the right word for this may be "schema". For applications and models (rather than data analysis), these sort of conventions can be super-valuable. I like the idea of declarative spec that can be validated.

Just googling around, I came up with pandas-validator: https://github.com/c-data/pandas-validator

[max-sixty]

I think the right word for this may be "schema"

Right! 🤦‍♂️

Just googling around, I came up with pandas-validator

Interesting, thanks.

Do you think this fits into a 'function which validates', rather than a Mypy-like type annotation? I think ideally there would be a representation of the schema that could work with both, so maybe this isn't the important question atm.

[max-sixty]

And let me know if there are already textual schema definitions from other libraries that you think are good, before we go and build our own (we don't work with any netCDF-like files so don't have that context)

[shoyer]

ncdump -h (xarray.Dataset.info()) is one existing schema of sorts, but it's hardly machine readable.

[benbovy]

Somewhat related to this issue, I have implemented in xarray-simlab some logic to validate xarray.Variable objects (dimensions, dtype, etc.). See this base class and some sub-classes here. I use that in a way which is quite similar to pandas-validator(i.e., using class attributes).

I'm currently in the process of refactoring this using attrs, which supports both validator functions and type annotations. Not sure how to use the latter for xarray objects, though (BTW I wasn't aware of python/typing#513, good to know!!).

I agree that it would be nice to have a more generic way to describe xarray objects that can be reused in many contexts.

[max-sixty]

@benbovy That looks v interesting. I think at the moment it would require a bit of work to validate normal xarray objects, is that right? (I'm looking at the __init__, which doesn't take the traditional args supplied to a Variable - tell me if I'm misreading it)

Separately - I didn't know about the project but looks awesome. Do we have a list of projects that integrate xarray? Let's start one somewhere if not @pydata/xarray ?

[benbovy]

@maxim-lian you're right. In this case xsimlab.Variable is a different concept than xarray.Variable, despite that they both have the same name. The former is tight to the modelling framework while the latter is only used for simulation inputs and outputs in xarray-simlab.

Do we have a list of projects that integrate xarray?

There is an ongoing discussion in #1850 about having something like xarray-contrib (likely a github organization).

[max-sixty]

The commentary in https://github.com/python/typing/issues/513, and @shoyer 's doc https://docs.google.com/document/d/1vpMse4c6DrWH5rq2tQSx3qwP_m_0lyn-Ij4WHqQqRHY/edit#heading=h.rkj7d39awayl are good & growing

I'll close this as I think riding on those coattails - with the addition of names and Datasets as containers - makes the most sense.

(though reopen if we think there's something we could productively do separately)

[JackKelly]

I'm really interested in a machine-readable schema for xarray!

Pandera provides machine-readable schemas for Pandas and, as of version 0.7, panderas has decoupled pandera and pandas types to make pandera more useful for things like xarray. I haven't tried pandera yet but I plan to do some experiments soon.

[shoyer]

Pandera provides machine-readable schemas for Pandas and, as of version 0.7, panderas has decoupled pandera and pandas types to make pandera more useful for things like xarray. I haven't tried pandera yet but I plan to do some experiments soon.

Awesome -- would love to hear how this goes!

[JackKelly]

OK, I think pandera isn't the way forwards because it appears very tighly coupled to Pandas (so, for example, I don't think it's possible to use pandera with n-dimensional arrays).

But Pydantic looks promising. Here's a very quick coding experiment showing one way to use pydantic with xarray... it validates a few things; but it's not super-useful as a human-readable specification for what's going on inside a DataArray or Dataset.

[rabernat]

But Pydantic looks promising

Big :+1: to this.

[andersy005]

• xref the more recent issue: https://github.com/pandera-dev/pandera/issues/705 which aims to implement a pandera.xarray module within pandera

[jhamman]

Related to the Pandera integration, we are prototyping the xarray schema validation functionality in the xarray-schema project.

[kubaraczkowski]

Does this project do (part of?) what's needed? +1 on making xarrays with explicit 'structure' !