data structure

[mikapfl]

As we discussed, a good data structure which is flexible enough for future use cases and at the same time comfortable to use needs to be the foundation for further work.

For that, I would like to evaluate "candidates" somewhat methodically, but I don't really have a plan how, yet.

[mikapfl]

As a first step, I will collect possible candidate data structures here, please chime in with your suggestions.

[mikapfl]

pandas DataFrame in traditional tidy format:

• the index is meaningless (row numbers)
• each column contains one variable
• each observation forms a row
• each type of observational unit forms a DataFrame
• columns ordered from fixed variables (left) to measured variables; within each group, alphabetical order
• rows ordered using the leftmost variable, breaking ties with variables further right

Even given these constraints, there are still some variables in the data representation:

• duplicate information (like countrycode and full countryname) in a single DataFrame vs. multiple DataFrames (only countrycode in the DataFrame, second DataFrame maps countrycodes to countrynames)
• units as composite types (e.g. pint) vs. units as a variable

Because the data structure relies on multiple, possibly joinable DataFrames, there is a need for a container, which is not yet defined.

[mikapfl]

A variation of the previous one would be the same format in principle, but all fixed variables are part of a MultiIndex. The semantics of MultiIndex are sufficiently different that I think it makes sense to treat this as a separate format.

[mikapfl]

xarray DataArray:

• all fixed variables are dimensions and coordinates
• space is "allocated" for one data point of observation
• each observation gets their own DataArray

A collection of DataArrays with shared dimensions/coords, i.e. where the coords of each individual DataArray is a subset of the shared coords, can form a Dataset.

Because not all DataArrays may share dimensions/coords, there is possibly a need for a container, which is not yet defined.

There are, again questions around representation of units.

[mikapfl]

scmdata TimeSeries:

• wrapper around xarray DataArray
• as the name suggests, Time is a special metadata
• has accessors for long format (not sure if strictly tidy)

A collection of TimeSeries can be held in a ScmRun object.

[mikapfl]

scmdata ScmDataFrame:

• own implementation with several filtering/etc functions
• is already deprecated in scmdata: https://github.com/openscm/scmdata/pull/59 , so I would consider it out of the race already at this point because of that.

[mikapfl]

pyam IamDataFrame:

• wrapper for two pandas DataFrames
• filtering functions etc. to keep metadata in sync with data

[mikapfl]

postgresql tables:

• tables in long / tidy format
• collection of tables in a single namespace
• supports integrity constraints between tables, in particular foreign key constraints
• unit support is rudimentary at best
• needs an online database
• integration of python functions available via PL/python

[mikapfl]

a variation of postgresql tables would be sqlite tables:

• also long format / tidy, with integrity constraints
• no unit support
• does not need an online database, available in python stdlib

[mikapfl]

datatoolbox Datatable

• subclasses pandas DataFrame
• adds some automatic unit handling based on pint
• adds handling and automatic propagation of metadata.

To manage a collection of Datatables, there is a custom database management system backed by git.

[mikapfl]

frictionless data Data Package

• based on json and csv
• own ecosystem of processors
• integrates with CKAN

[mikapfl]

datalad

• dataset management system
• based on git and git-annex
• can track and query metadata
• also has provisions for reproducing data and tracking provenance
• agnostic to the structure of datasets, as long as they can be tracked as files

[mikapfl]

Results are summarized here: https://github.com/pik-piam/primap/blob/main/data%20structures/datalibraries_evaluation_results.pdf