search

OpenEnergyPlatform / data-preprocessing

Repository for data formatting, import of data, data and metadata review, and data curation.
GNU Affero General Public License v3.0
10 stars 7 forks source link

Review: fh_iee_trafo_fw #75

Closed Richard-Schmitz closed 2 years ago

Richard-Schmitz commented 3 years ago

Issue description

I'm submitting an open dataset with a corresponding OEMetadata string for review. The dataset shows the results from project "Fh IEE Trafo FW" ("Pathways of transformation in the heating sector") of Fraunhofer IEE. It contains a total of 10 different input tables as well as 15 output tables for each of the 4 scenarios.

Workflow checklist

  1. GitHub

    • [x] I have submitted this issue to have metadata and data review documented (Issue #74)
    • [x] Create a new review-branch and push OEMetadata to new branch (review/project_nameofdata#NR)

  2. OEP

    • [x] Upload data to the OEP in schema model_draft (see upload tutorial)
    • [x] Link URL of data in this issue (model_draft.project_nameofdata)

  3. Start a Review

    • [x] Start a pull request (PR) from review-branch to master
    • [x] Assign a reviewer and get in contact

  4. Reviewer section

    • [x] A reviewer starts working on the issue
    • [x] Review data license
    • [x] A reviewer finished working on this issue (and awarded a badge)
    • [x] Update metadata on table
    • [x] Data moved to its final schema
    • [x] Add OEP tags to table
    • [x] Merge PR and delete review-branch
    • [x] Document final links of metadata and data in issue description
    • [x] Close issue

Metadata and data for review

Here are the links to my data and metadata. Naming follows the pattern model_draft.project_nameofdata:

Metadata: PR: https://github.com/OpenEnergyPlatform/data-preprocessing/pull/74

v1.5.1 https://github.com/OpenEnergyPlatform/data-preprocessing/tree/review/fh_iee_trafo_fw/data-review/75-fh_iee_trafo_fw

Data:

Input data: 01) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_input_biomassepotential 02) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_input_cop_zeitreihen 03) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_input_ee_potentiale 04) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_input_emissionen 05) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_input_fahrzeugpotentiale 06) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_input_kuppelleistungen 07) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_input_laender 08) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_input_rahmendaten 09) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_input_technologieparameter 10) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_input_waermepotentiale

Output data: 01) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_hohe_sani_emissionen 02) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_hohe_sani_endenergie 03) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_hohe_sani_fw_bilanz 04) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_hohe_sani_fw_erzeugung 05) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_hohe_sani_gasf_kws_bilanz 06) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_hohe_sani_gebaeude 07) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_hohe_sani_h2_bilanz 08) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_hohe_sani_h2_erzeugung 09) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_hohe_sani_interkonnektoren 10) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_hohe_sani_speicher 11) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_hohe_sani_strom_bilanz 12) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_hohe_sani_strom_erzeugung 13) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_hohe_sani_systemkosten 14) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_hohe_sani_verkehr 15) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_hohe_sani_zeitreihen 16) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_mod_sani_emissionen 17) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_mod_sani_endenergie 18) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_mod_sani_fw_bilanz 19) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_mod_sani_fw_erzeugung 20) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_mod_sani_gasf_kws_bilanz 21) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_mod_sani_gebaeude 22) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_mod_sani_h2_bilanz 23) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_mod_sani_h2_erzeugung 24) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_mod_sani_interkonnektoren 25) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_mod_sani_speicher 26) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_mod_sani_strom_bilanz 27) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_mod_sani_strom_erzeugung 28) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_mod_sani_systemkosten 29) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_mod_sani_verkehr 30) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_viel_dez_bio_mod_sani_zeitreihen 31) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_hohe_sani_emissionen 32) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_hohe_sani_endenergie 33) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_hohe_sani_fw_bilanz 34) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_hohe_sani_fw_erzeugung 35) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_hohe_sani_gasf_kws_bilanz 36) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_hohe_sani_gebaeude 37) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_hohe_sani_h2_bilanz 38) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_hohe_sani_h2_erzeugung 39) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_hohe_sani_interkonnektoren 40) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_hohe_sani_speicher 41) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_hohe_sani_strom_bilanz 42) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_hohe_sani_strom_erzeugung 43) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_hohe_sani_systemkosten 44) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_hohe_sani_verkehr 45) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_hohe_sani_zeitreihen 46) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_mod_sani_emissionen 47) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_mod_sani_endenergie 48) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_mod_sani_fw_bilanz 49) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_mod_sani_fw_erzeugung 50) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_mod_sani_gasf_kws_bilanz 51) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_mod_sani_gebaeude 52) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_mod_sani_h2_bilanz 53) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_mod_sani_h2_erzeugung 54) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_mod_sani_interkonnektoren 55) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_mod_sani_speicher 56) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_mod_sani_strom_bilanz 57) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_mod_sani_strom_erzeugung 58) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_mod_sani_systemkosten 59) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_mod_sani_verkehr 60) https://openenergy-platform.org/dataedit/view/model_draft/fh_iee_trafo_fw_wenig_dez_bio_mod_sani_zeitreihen

Reviewed and published metadata and data

Final naming and location of the data and metadata after the review are as follows: schema.tablename

christian-rli commented 3 years ago

I had to make a few changes so omi could parse the string. Mainly these were replacing "null" with null, "none" with null and deleting empty contributors info. One thing affecting content needed a change: the start and end date of the timeseries weren't datee. They were strings of two dates which both didn't conform to ISO 8601. I changed these to the earliest date for start and the latest date for end. Please let me know if that's a broad mischaracterisation. I'm assuming that the files describe two timeseries in different files? Other than that the metadata are good to go from my perspective. (On a side note: Yay vim for allowing search and replace in a long list of files!)

Richard-Schmitz commented 3 years ago

Thanks for the review, your changes work out fine. Concerning the timeseries, it is always one spreadsheet containing data for both 2030 and 2050, so the earliest date for start and the latest date for end are correct now.

christian-rli commented 3 years ago

@MGlauer or @meisam-booshehri I'm trying to update the metadata on these corresponding tables, but the OEP tells me that I don't have permission. Neither the webinterface nor the API allow write access. Do you have any idea why that is? Thank you for any information.

Traceback (most recent call last): File "fh-meta.py", line 19, in <module> cl.update_metadata(metadata=data) File "/home/christianh/anaconda3/lib/python3.7/site-packages/oep_client/client.py", line 133, in update_metadata self.request("POST", url, jsondata=metadata) File "/home/christianh/anaconda3/lib/python3.7/site-packages/oep_client/client.py", line 53, in request raise Exception(err) Exception: {'detail': 'You do not have permission to perform this action.'}

MGlauer commented 3 years ago

Well, probably because you do not have write permissions. The admin of the table - most probably the creator - has to grant those to you.

christian-rli commented 3 years ago

Thank you @MGlauer . It was my impression that tables created via the api are all created by the same user and can therefore be modified(?). It does sound wrong when writing it out though.

Has this changed? I believe, I used to be able to write metadata to other people's tables. Also, how would one grant write access?

... I checked a few of my tables and it looks like I don't have write access to those either. Is my database user connected to my api user? I will need some sort of access in any case for moving the table to its final schema.

@Richard-Schmitz: Could you grant me write access, please? My OEP user name is "Christian Hofmann" And asking out of interest: how were the tables created?

christian-rli commented 3 years ago

Never mind the question on how to grant permission rights. The OEP has a big red button for that which I never noticed. Also I found a table that I created and I have access to, so it seems that just database user is not connected to my oep user account and I have created most of my tables with the former.

christian-rli commented 3 years ago

@MGlauer I moved the tables to the schema "scenario" and now all tags are lost. This happened before in #27 . I recall that I had to run a command via sql to bring them back, but it wasn't documented anywhere. Could you let me know (again), please? (This time I shall document it properly at an appropriate location)

This shouldn't have to happen. I suppose when moving tables via API this wouldn't happen, but API users aren't allowed to move tables, as far as I am aware. There should be a fix for this in the long term.

steull commented 2 years ago

I have updated all metadata from version 1.4 to version 1.5.1. I noticed that the characters for umlauts (for example) are not parsed. I assume these are not ASCII characters? I haven't changed that yet. I also noticed that all column names are capitalized. Here, the OEP requires all words to be lower case. (I assume we can change that so you don't have to create the tables and upload the data again) Furthermore, I changed the source to "null" since "own project" is not a valid source. I also made minor adjustments like spatial resolution and timeseries resolution.

Tables moved to scenario schema and all tags are added to the table. Review process is finished.