search

OpenEnergyPlatform / data-preprocessing

Repository for data formatting, import of data, data and metadata review, and data curation.
GNU Affero General Public License v3.0
10 stars 7 forks source link

review/ksz2050 #27

Closed wingechr closed 4 years ago

wingechr commented 5 years ago

We have uploaded the first three tables for the ksz2050 scenario and submit them to the review process:

We added the meta data json files in a new branch: https://github.com/OpenEnergyPlatform/data-preprocessing/tree/review/ksz2050 (but no pull request yet).

l-emele commented 5 years ago

We have uploaded further tables for the ksz2050 scenario to be reviewed:

All in the review/ksz2050 branch: https://github.com/OpenEnergyPlatform/data-preprocessing/tree/review/ksz2050

christian-rli commented 5 years ago

I had a first look at the metadata. Unfortunately the metadata-structure does not conform to the latest standard (our fault for not keeping you up to date). I'm going to update the structure to "OEP-1.4.0" and then see further.

christian-rli commented 5 years ago

Specifically, the following issues are addressed in the restructuring:

han-f commented 5 years ago

We updated the metadata another time and added publisher logo, corrected contact email etc. There is a pull request now (https://github.com/OpenEnergyPlatform/data-preprocessing/pull/39 ) since we think the data including metadata is now final and can be moved to the scenario schema.

christian-rli commented 5 years ago

I moved the tables to the scenario schema, but metadata aren't visible yet. Version 1.4 is not fully supported yet. Maybe I can convert it back to the old version to have the metadata show in the mean time.

han-f commented 5 years ago

Is omi not yet able to do this? From my perspective it would be very useful if the metadata could be shown very soon. It includes a phrase about the funding, which needs to be shown in any case. I am thus wondering whether there is an easy way to only include that phrase for now and then make the 1.4 metadata shown once the platform is ready.

han-f commented 5 years ago

Many thanks for moving the data. In the process, all tags we assigned got lost. At least they are not attached to the tables any longer. I have opened a separate ticket for this: https://github.com/OpenEnergyPlatform/oeplatform/issues/426

christian-rli commented 5 years ago

Sorry about the lost tags @han-f . Hopefully we can restore them.

omi ist able to understand version 1.4, but support/implementation into the oep is not complete. This is a pending issue already called for here, but I just realized there is no actual issue for this. So I created a new one here: https://github.com/OpenEnergyPlatform/oeplatform/issues/427 . Sorry @MGlauer :)

The older version of our metadata (1.3) does not have a "fundingAgency" or equivalent key. A temporary solution would be to add the funding phrase into the description of the older version. I tried to parse the metadata into version 1.3, but ran into new issues.

han-f commented 5 years ago

Thanks @christian-rli. The tags are back, many thanks! Interestingly, however if I click a tag in the most popular tags sidebar - I do not get any results.? If it worked out to have the funding phrase in the description this would be great too.

han-f commented 4 years ago

@christian-rli : any news on adding the funding phrase into the description field of metadata version 1.3?

christian-rli commented 4 years ago

Sorry for the wait @han-f . Unfortunately converting backwards still has some issues. I decided to upload the string in version 1.4.0 to all tables. Now it doesn't look pretty, but at least the information is there. When support for 1.4.0 finally arrives these will be the first tables to work properly (and a good reference point for the implementation as well).

l-emele commented 4 years ago

I just had an look on this issue. This issue is now pending five months and to me it is not exactly clear what changes in out metadata are needed that we can finalize the review of the metadata and create a pull request.

@christian-rli: Could you please exactly specifcy the changes needed in our metadata (e.g. as a to do list)?

christian-rli commented 4 years ago

Sorry if this issue seems pending @l-emele . No changes need to be done. The strings look good and were uploaded to the OEDB. The only remaining issue is that they don't look nice on the OEP yet. No strings of version 1.4 do up until this moment. This issue was addressed in https://github.com/OpenEnergyPlatform/oeplatform/issues/427 , so once the next release is out, everything should look fine, as well. I kept this issue open as a reminder to see if it actually does. If you prefer, we can close this.

han-f commented 4 years ago

Does that mean that the data in the database is correct, but what is currently displayed on OEP is wrong? I only ask because one thing is different between what we submitted here and what is displayed currently on OEP: the email address for contact.

l-emele commented 4 years ago

On the OEP also in our metadata file it shows oedb@oeko.de. That is the correct one.

han-f commented 4 years ago

Oh interesting - on the OEP the (not pretty) metadata showed szendb@oeko.de for me. Let me check if this was a browser hiccup

han-f commented 4 years ago

On the OEP the email address at the beginning is correct, the one under contributors is still szendb@oeko.de, can this easily be corrected for all KSZ tables?

jh-RLI commented 4 years ago

What @christian-rli meant is that the representation of the metadata string in version 1.4 on the oep does not look good yet. The string is displayed as plain text and not formatted. This will change after the relaunch.

jh-RLI commented 4 years ago

To change the content of the metadata on the oep the metadata files on github have to be updated. @christian-rli can then append the corrected version to the tables.

han-f commented 4 years ago

Thanks @jh-RLI and yes I know I need to update them here. I did this now by opening each file once and changing it. I just wondered above whether it would be possible to do it more efficiently than opening each single file, as there are many. @christian-rli could you please append the updated version to the tables. I have changed the email address under contributors to the correct one. Apologies for not adding this as a header to each commit message.

jh-RLI commented 4 years ago

@han-f oh okay. Unfortunately, I do not know a better solution than search and replace. Maybe an idea for another tool that can adjust multiple metadata files at once.

christian-rli commented 4 years ago

The updated versions are now attached to the tables @han-f . Thank you @Ludee for being my remote sql-console and fixing my little errors :)

han-f commented 4 years ago

Thanks. That probably means this ticket can be closed.