Review: Metadata UMAS oemof_B3 #103

[chrwm]

Closes #103

This branch will implement metadata for UMAS input data.

Collection of mistakes that occurred with this metadata for SIROP

metadata

• [x] json formatting error, e.g. missing , or ( etc.
• [x] csv filename and metadata filename cannot be matched @jh-RLI I match them by inserting the same filename in the resources as well as name, title, id fields; maybe specify technical minimal requirements
• [x] column name in metadata doesn't match column header in csv and vice versa
  [Matching of csv column header and column name in metadata is now automated with similarity check - feature will be released] https://github.com/sedos-project/open_annotation_tool/issues/3
• [x] column types were wrong -> clear guidelines with examples needed
• [x] people couldn't differ between column type text -> t002 and in int
• [x] sometimes a column type was declared with more than one type, such as int and text -> clear information that one column can have only one type -> when multiple types in one column than text -> or get a better dataformat

errors regarding technical processing

• [x] missing type for column
• [x] missing field-dict for an entire column in metadata (-> column is not represented in metadata)
• [x] missing description field for column

@jh-RLI please check regarding oem2orm requirements vs OEMetaBuilder

data

• [x] column header not postegresql conform (e.g. 99c600f)
  [Making column header postgresql-conform is now automated - feature will be released] https://github.com/sedos-project/open_annotation_tool/issues/3
• [x] 1st line is no column header, but documentation of data (e.g. for this file a5c41ba)
• [x] csv separator is mixed among several files > could cause trouble with oem2orm during automatic processing

[chrwm]

Hi @hedwiglieselotte @r0gga, thank you for creating the first metadata version! Your commit messages look good! Thank you for your work, and congratulations on creating your first metadata! @r0gga It seems sometimes you researched and provided a source when it was missing, which was actually not necessary because it is their responsibility to manage and deliver their sources. In doing so, you saved the UMAS team some work. @Stefanie08 will take over from here and will review the metadata content-wise.

[Stefanie08]

Hello @r0gga and @hedwiglieselotte, thank your for creating the metadata. This is a great help. Could you support me on this task again? I would like to ask you to check your metadata again and correct the value for the key: type. Examples are "text", "serial" or "integer". For "text" it should be the term "string". If you are not familiar with the terminology there is an overview in Table 1 on this website https://www.sivakids.de/python-datentypen/.

@r0gga: It would be nice, if you can fill in your e-mail address into the contribution part.

If there are any questions feel free to text me or have a look on demands.csv, which I have already reviewed. Thank you!

[jnnr]

This is the list of the metadata files with the person assigned that knows most about the respective data. @MaGering, @monika-o, @Stefanie08: Please have a quick check for those datasets you are assigned to (if you haven't already). We will not provide every value, but the most general entries should be correct. Please add to the "contributors". If you are lacking write permissions, please get in touch with me.

• 2021-05-18_pv_agriculture_brandenburg_kreise_epsg32633.json SH
• 2021-05-18_pv_road_railway_brandenburg_kreise_epsg32633.json SH
• 2021-05-18_wind_brandenburg_kreise_epsg32633.json SH
• building_class.json @MaGering
• capacities.json @jnnr
• carrier_costs.json @jnnr
• conventional_power_plants_DE.json @jnnr
• costs_efficiencies.json @monika-o
• demands.json
• distributions_households.json @MaGering
• DIW_Hydro_availability.json
• el_gas_constraint.json @Stefanie08
• emission_constraints.json @Stefanie08
• holidays.json @MaGering
• id_region_BB_B.json
• import_export_prices.json @jnnr
• lookup_table_brandenburg_kreise.json SH
• ninja_pv_country_DE_merra-2_nuts-2_corrected.json SH
• ninja_wind_country_DE_current_merra-2_nuts-2_corrected.json SH
• potentials.json SH
• res_SimBEV_Berlin_2015.json SH
• res_SimBEV_Berlin_2017.json
• res_SimBEV_Berlin_2019.json
• res_SimBEV_Brandenburg_2015.json
• res_SimBEV_Brandenburg_2017.json
• scalars_methanation.json @jnnr
• ts_load_ch4.json @jnnr
• ts_load_h2.json @jnnr
• weatherdata.json @MaGering

[chrwm]

open tasks:

• [x] delete empty foreignKey for upload
• [ ] fix id and put umas_ at beginnig of each table name
• [x] make csv-and metadata columns postrgresql-conform > use datahelper
• [x] along the way fix JSON format errors
• [ ] fix missing/wrong metadata col-name
• [ ] fix wrong col-types

[jh-RLI]

General info (also looking at the json formatting errors) "close" future - omi will be able to validate the metadata json based on the information provided in this schema. That will enable metadata validation for:

• json fomat
• value type (e.g."key":"value"<- this type is checked ig string or none, other types like datetime format, integer, .... are also checked.)
• Missing keys
• maybe key order
• ... other things that we want to validate can be added Working on this here: https://github.com/OpenEnergyPlatform/omi/pull/63

[jh-RLI]

Regarding the list of technical errors in the metadata - I think this is it we should try to run some of the metadata with oem2orm. The metaBuilder should also work - but we cant fully test this until the jsonschmea validation is release (what i described in my other comment) but lookin at this it seems fine to me.

[chrwm]

branch is not deleted for sirop documentation