Revision of the OEMetadata structure to improve usage and technical correctness

[jh-RLI]

@wingechr We talked about this in the project meeting last week. Could you please add a brief sketch of what you mean so we can discuss it and make it available to others?

I think if you could quickly change the structure of the Oemetada example to what you explained to me, that would help me and others understand it - I thought I understood, but I forgot what key you were talking about :) I think it was about resources or schema?

--> https://github.com/OpenEnergyPlatform/oemetadata/blob/develop/metadata/latest/example.json

[wingechr]

OLD

• json metadata for table:

  {
  "resources": [
      {
          "name": "<TABLE_NAME>"
          "schema": {...}
      }
  ],
  <METADATA>
  }

• json metadata for datapackage: DOES NOT EXIST

PROPOSED NEW

• json metadata for table (aka resource):

  {
  "name": "<TABLE_NAME>",
  "path": "<URL>",
  "schema": {...},
  <METADATA>    
  }

• json metadata for datapackage:

{
    "name": "<BUNDLE_NAME>",
    "resources": [
        {
            "name": "<TABLE_NAME_1>",
            "path": "<URL>",
            "schema": {...},
            <METADATA>    
        },
        {
            "name": "<TABLE_NAME_2>",
            "path": "<URL>",
            "schema": {...},
            <METADATA>    
        },
        ...
    ]
}

• name and path are required (by specs) for resource,
• name and resources are required for datapackage

[henhuy]

Agree with your proposal!

[wingechr]

sorry, one more correction: in the bundled datapackage.json that presumably is part of a zipped datapackage in a download, the resource path must be changed to the now local csv file that contains the table data. There can also be some more additional metadata values (like csv encoding) that don't make sense for data in the database, but for it's physical representation in a file when downloaded:

{
    "name": "<BUNDLE_NAME>",
    "resources": [
        {
            "name": "<TABLE_NAME_1>",
            "path": "data/<PATH/TO/CSV.csv>,
            "encoding": "utf-8",
            "schema": {...},
            <METADATA>    
        },
        {
            "name": "<TABLE_NAME_2>",
            "path": "data/<PATH/TO/CSV.csv>,
            "encoding": "utf-8",
            "schema": {...},
            <METADATA>    
        },
        ...
    ]
}

[jh-RLI]

I added an example presenting what my understanding of the changes is.

• ressource
• datapackage

One more thing, we have to figure out where to put the keys that have been stored in the old resources.

"profile": "tabular-data-resource",
"name": "model_draft.oep_metadata_table_example_v152",
"path": "http://openenergyplatform.org/dataedit/view/model_draft/oep_metadata_table_example_v152",
"format": "PostgreSQL",
"encoding": "UTF-8", 

• name would duplicate the name of the metadata
• path is kinda duplicated with id
• profile ...

Other tasks after this one is completed.

• Update the schema! and other oemetadata files
• Breaking changes for OMI (new structure needs to be implemented)

[wingechr]

just to be clear: the metadata is different when it is on the platform and when it is in a downloaded zip package.

most notably:

• path: on the server, it can be the url (and is redundant with the id) or it can be dropped entirely. BUT on in a local zip package it MUST be the relative path to the csv file (e.g. data/table.csv)

• profile, encoding, format, ...: on the server, they can (and should) be omitted entirely, because this is information only relevant for a local csv file.

as an example: i could upload a csv file with encoding iso-8556-1, but in the postgres database it will always be utf-8, so the encoding information becomes irrelevant. if i download it again though, the encoding of the csv file (once created) becomes relevant for the user again.

[jh-RLI]

True, we should add functionality to the OEP metadata upload and download that removes and adds this information? Then the main change to the current oemetadata spec. would just be the removed resources key.

[wingechr]

yea. i think like this:

• remove profile, encoding, ... from the specs and drop on upload
• path should stay part of the specs, because it is part of frictionless specs, but programmatically set it to url on upload
• name should be programmatically set to the table name
• on normal metadata download: don't change anything
• only in package download:
  • create new datapackage.json
  • get metadata for all the requested tables and put them into the resources array
  • programmatically add profile, encoding, for the resources csv data
  • programmatically change path in each resource to the csv file