spine-tools / Spine-Database-API

Database interface to Spine generic data model
https://www.tools-for-energy-system-modelling.org/
GNU Lesser General Public License v3.0
6 stars 5 forks source link

How metadata should be used? #329

Closed soininen closed 5 months ago

soininen commented 6 months ago

I would like to start discussing what to do with the metadata documentation. I did not partake in any metadata discussions during the Spine project so let me know if I am heading straight off the cliff here.

During the last stages of the Spine project, I salvaged the metadata description and result metadata description documents from the wiki of our old GitLab repository. These documents describe metadata as a JSON container inspired by datapackage.

While the container is in my opinion pretty nice, I am afraid JSON blobs might be a bit overwhelming to users unless we provide a comprehensive editor. Further, using the JSON containers, the metadata table would be filled with records like this:

name     value
----     -----
eodc-api <JSON container>
ref_data <JSON container>

My understanding is that the metadata table is actually supposed to look like this:

name     value
----     -----
author    A. Soininen
author    B. Soininen
date      1998-13-13T13:21
source    https://en.wikipedia.org

So, what to do with the metadata description documents? Do we want to keep the metadata containers? In that case we should add something more concrete into the documents, i.e. how to actually fill the metadata records. Or should the JSON containers be replaced by something simpler? In that case we should replace the current metadata documentation.

Let me know your thoughts!

jkiviluo commented 5 months ago

@soininen and me discussed this a bit. There are also benefits to a JSON blob (or maybe TOML for better readability). Having everything in one blob translates to less rows in the linking table between metadata and parameter values (or entities). To be discussed in the upcoming Leuven meeting.

soininen commented 5 months ago

We decided to keep the metadata tables as-is at Leuven. The JSON specifications in the documentation should be replaced with "Recommended metadata practices" page.