search

sentinel-energy / friendly_data

Data format to interoperate between models and frameworks
https://sentinel-energy.github.io/friendly_data/
Apache License 2.0
12 stars 2 forks source link

Support unit conversion within Friendly data #24

Open suvayu opened 2 years ago

suvayu commented 2 years ago

There are two kinds of units to deal with: physical units, and economic units. Domain specific physical units can be defined and maintained within friendly data using pint. For economic units, salamanca can be used.

Since units are often added as columns (it's also in the registry), adopting that is acceptable, although it inflates file size. An alternative would be to add the unit to the metadata of all relevant columns, similar to alias. However, that is essentially a "fork" of the frictionless spec, and would add a maintenance burden.

As for behaviour during conversions, when doing it in-memory using the Python API, it is simple, the unit conversion functions can do the conversion while reading a dataset into memory. There is no reason why the API cannot also work on dataframes/data arrays directly. When using the CLI, we can adopt the --export/--inplace modes of operation. The functionality can be a separate module, like tseries.

danielhuppmann commented 2 years ago

Let me also point you to the iam-units package (https://github.com/IAMconsortium/units), which is pint with a additional definitions for units common in energy systems modeling, like "ton of coal equivalent" and redefining kt to be "kiloton" instead of "knot". And iam-units also has a conversion module between emissions species (greenhouse gases) according to several global-warming-potential (GWP) metrics.

cc @khaeru @gidden

suvayu commented 2 years ago

@danielhuppmann thanks for pointing to iam-units, it slipped my mind :(. Friendly data can of course build on top of that.