Upload tutorial is out of date

areleu commented 1 year ago

Description of the issue

There are multiple issues in this particular tutorial: OEP Upload Process Data and Metadata

Broken link: https://github.com/OpenEnergyPlatform/academy/tree/production/upload/upload_template.ipynb
Broken link: https://github.com/OpenEnergyPlatform/academy/blob/feature/upload_process_tutorial/upload/OEP_Research_Data_Publishing_Guidebook.ipynb
"ModuleNotFoundError: No module named 'oem2orm'" error displaying, probably the parsing script is not working properly.
Uses environmental variable: "OEP_API_TOKEN", but in the code of the tool: oep_oedialect_oem2orm.py#L440 the environmental variable is called "OEP_TOKEN"
If the metadata file does not have foreign keys it will throw an error at oem2orm.create_tables(db, tables_orm)
The oem2orm api has a lot of hardcoded components, I think it should use omi as a backend to be compatible with all the versions of oem.

Ideas of solution

The tool is relatively intransparent, is it not possible to upload data and metadata using the API? If so, it would be nice to have some documentation on this, when clicking on the API link it brings me to the tutorials, but I think there should be somewhere some general API documentation with a list of commands. Unfortunately https://oep-data-interface.readthedocs.io/en/latest/api.html only tells how to upload data but no metadata and it is not very complete.

As for the python API, if I have a valid csv file with its associated metadata file. I would expect to have something like:

import oepapi as oep
import os
SCHEMA = "model_draft"
DATA = "my_awesome_data.csv"
METADATA = "my_awesome_data.json"

USER = "my_user"
TOKEN = os.environ["OEP_TOKEN"]

oep.publish(data=DATA, metadata=METADATA, schema=SCHEMA, user=USER, token=TOKEN)

If the metadata is validated with omi and the dataset is writtten in accordance with the metadata then the api should take care of building the table in the OEP. It should also take care of the connection in the background.

The API should be able to take pandas dataframes and dictionaries respectively, or that would be nice.

Context and Environment

Version used:
Operating system: Windows
Environment setup and (python) version: 3.10

Workflow checklist

[x] I am aware of the workflow in CONTRIBUTING.md

Ludee commented 1 year ago

Thank you for spotting and documenting theses bugs. Most of the tutorials are currently outdated and some functions do not work (anymore). The OEP developers will restructure all existing tutorials in the next weeks.

A the moment, there are multiple ways and tools to interact with the oedb. Not all work as expected and most is not documented properly:

OEP frontend
OEP-API
oedialect
oem2orm
oep-client

jh-RLI commented 5 months ago

I like the idea and also see the need to implement such an updated API to facilitate the upload process. Currently we need to focus on other areas of the oefamily.

I think we should eventually bundle the functionality of all tools in the oep client. I would also advise using the oep client to interact with the database.

Regarding the http-api documentation: we know that it is quite outdated and incomplete. It is currently on my list, but will also have to wait until I can make room for it. I plan to use swagger-ui and openAPI schemas to create full documentation for each available endpoint.

We will also tackle a complete revision of the omi tool.

han-f commented 3 weeks ago

This linked tutorial is not available any more under the link. Since we have worked a lot on the OEA and restructuring it, I close this issue for now.

OpenEnergyPlatform / academy