Data oep upload

[jnnr]

Closes #255

The script will enable an automated data upload to the OEP via the API.

The tables on the OEP are created from the metadata and later filled with the content from the csv-files. Thus, metadata (description of columns) for all tables need be in place before uploading the data itself.

The script should do this:

• Take as input the mapped results data of a specific scenario
• Create metadata for the results
• Upload the mapped results and metadata to the oep.
• Warn if data already exists
• Allow to update or delete data in model draft?

Test data uploaded: https://openenergy-platform.org/dataedit/view/model_draft?query=oemof_b3

TODO:

• [x] Create metadata (probably from default, filling in the scenario specific information)
• [x] Find out how it is possible to trigger the upload using snakemake.
• [x] Give information on how to provide oep credentials: In config/.secrets.yaml (make sure that it is mentioned in .gitignore)
• [x] Show script header in docs
• [x] Finalize script header
• [x] Keep metadata order
• Set metadata
  • [x] Title
  • [x] Keywords (tags)
  • [ ] IsAbout (ontological annotation)
  • [x] Correct scenario name (Names must consist of lowercase alpha-numeric words or underscores and start with a letter and must be have a maximumlength of 50)
• [x] Need to write values to resources.

[jnnr]

We could define the snakemake rule like this:

inputs:
   results/{scenario}/b3_results/data
outputs:
   results/{scenario}/b3_results/metadata

[jnnr]

Next steps:

update: Moved to first post.

[jnnr]

I get this error

Traceback (most recent call last):
  File "scripts/upload_results_to_oep.py", line 209, in <module>
    oem2orm.omi_validateMd(metadata)
  File "python3.8/site-packages/oem2orm/oep_oedialect_oem2orm.py", line 387, in omi_validateMd
    OmiParser.parse(data)
  File "/python3.8/site-packages/omi/dialects/oep/parser.py", line 258, in parse
    start=parse_date_or_none(inp_timeseries.get("start")),

[jh-RLI]

I know about this error, just have to look into this again. Also I'm still working on the omi release - this will also make things easier I think.

[jh-RLI]

@jnnr please try again. I have updated oem2orm on pypi and added a new code module called oep_compliance.py - long story short the function omi_validateMd() is now updated and can handle oemetadata v1.5 and v1.4.

[jh-RLI]

@jnnr you can now also use the jsonschema validation implemented in omi. This ensures that your metadata matches the corresponding (version-specific) schema of oemetadata. Install omi v0.1.0: Validation is implemented in the parser. The easiest way to use it is as follows:

# You can import the JSONParser directly like this:
import json
from omi.dialects.oep.parser import JSONParser

with open("tests/data/metadata_v15.json", "r", encoding="utf-8") as f:
    metadata = json.load(f)

# validate and create a report.json containing all errors in ./reports/
parser = JSONParser()
parser.validate(metadata)

# check if your metadata is valid for the given schema and returns true or false
schema = ... get a schema or import form oemetadata module
parser.is_valid(metadata, schema)

[jh-RLI]

You should consider updating your metadata template - now it is also possible to use the oemetadata as pypi module and import the json template. But for now you could just update the oemetadata.json file using this template for v1.5.2.