Create metadata for existing datasets

nesnoj commented 3 years ago

Now there're a howto and examples on metadata #176. Now it's up to all of us:

Everyone, please

Check the DB table list below and
- Tick the column Metadata needed? if you think this is a table which needs further explanation (at least all tables we're up to publish on the OEP #367 do need metadata)
- Add your nick for datasets you're responsible for
- Tick Done! if you're done, even it has not been merged to dev yet
Create metadata for your tables and make sure they comply with the OEP 1.4.1 standard. Read the instructions how to do so, see finished datasets below for examples.

Notes

There're some helpers (e.g. a bunch of standard licenses), see instructions. See examples on how to use them.
You do not have to set the fields id and publicationDate as they will be set in a final metadata task #451. But you have to set them to make OMI :older_woman: happy, example.
A new task #451 will be created which will be executed at the end of the pipeline that checks all metadata against the OEP 1.4.1 standard via OMI tool (regardless of this final check, please make sure your metadata string adheres to the standard).

Examples: There're OMI-approved metadata for datasets of 3 different tasks:

If you have further questions or remarks, please drop me a line...

List of DB tables

table_schema	table_name	Metadata needed?	Responsible person	Done!
boundaries	egon_map_mvgriddistrict_vg250	YES	@ClaraBuettner @IlkaCu
boundaries	egon_map_zensus_buildings_filtered	YES (Really?)
boundaries	egon_map_zensus_grid_districts	YES	@ClaraBuettner @IlkaCu
boundaries	egon_map_zensus_vg250	YES	@ClaraBuettner @IlkaCu
boundaries	vg250_gem	YES	@nesnoj	X
boundaries	vg250_gem_clean	YES	@nesnoj
boundaries	vg250_gem_population	YES	@nesnoj	X
boundaries	vg250_krs	YES	@nesnoj	X
boundaries	vg250_lan	YES	@nesnoj	X
boundaries	vg250_rbz	YES	@nesnoj	X
boundaries	vg250_sta	YES	@nesnoj	X
boundaries	vg250_vwg	YES	@nesnoj	X
boundaries	bkg_vg250_statistics_view
boundaries	egon_map_zensus_climate_zones	YES
boundaries	inspee_saltstructures
boundaries	egon_map_zensus_mvgd_buildings	YES	@nailend
boundaries	egon_map_zensus_vg250
boundaries	egon_map_zensus_buildings_filtered_all
boundaries	egon_map_zensus_buildings_residential
boundaries	egon_map_zensus_weather_cell	YES
demand	egon_demandregio_cts_ind	NO
demand	egon_demandregio_hh	NO
demand	egon_demandregio_osm_ind_electricity	NO
demand	egon_demandregio_sites_ind_electricity	NO
demand	egon_demandregio_timeseries_cts_ind	NO
demand	egon_demandregio_wz	NO
demand	egon_demandregio_zensus_electricity	YES	@ClaraBuettner
demand	egon_district_heating_areas	YES	@ClaraBuettner
demand	egon_etrago_electricity_cts	YES
demand	egon_etrago_heat_cts	YES
demand	egon_etrago_electricity_households	NO
demand	egon_hotmaps_industrial_sites	NO
demand	egon_household_electricity_profile_in_census_cell	YES	@nesnoj
demand	egon_industrial_sites	YES	@IlkaCu
demand	egon_map_zensus_district_heating_areas	YES	@ClaraBuettner
demand	egon_osm_ind_load_curves	NO
demand	egon_peta_heat	YES	@ClaraBuettner
demand	egon_schmidt_industrial_sites	NO
demand	egon_seenergies_industrial_sites	NO
demand	egon_sites_ind_load_curves	NO
demand	iee_household_load_profiles (check PR523)	YES	@nesnoj
demand	egon_cts_heat_demand_building_share	YES
demand	egon_cts_electricity_demand_building_share	YES
demand	egon_building_electricity_peak_loads	YES
demand	egon_loadarea	YES
demand	egon_timeseries_district_heating	YES
demand	egon_building_heat_peak_loads
demand	egon_household_electricity_profile_of_buildings	YES
demand	egon_osm_ind_load_curves_individual	YES
demand	egon_sites_ind_load_curves
demand	egon_sites_ind_load_curves_individual	YES
demand	egon_etrago_electricity_cts_dsm_timeseries
demand	egon_osm_ind_load_curves_individual_dsm_timeseries
demand	egon_heavy_duty_transport_voronoi
demand	egon_daily_heat_demand_per_climate_zone	YES
demand	egon_hp_capacity_buildings	YES
demand	egon_ev_count_registration_district
demand	egon_ev_count_municipality
demand	egon_ev_count_mv_grid_district
demand	egon_ev_pool
demand	egon_ev_trip	YES
demand	egon_ev_mv_grid_district	YES
demand	egon_ev_metadata	YES
demand	egon_heat_idp_pool	YES
demand	egon_heat_timeseries_selected_profiles	YES
grid	egon_ehv_substation	YES	@IlkaCu
grid	egon_ehv_substation_voronoi	NO
grid	egon_etrago_bus	YES	@ClaraBuettner @AmeliaNadal
grid	egon_etrago_bus_timeseries	YES	@ClaraBuettner @AmeliaNadal
grid	egon_etrago_carrier	YES	@ClaraBuettner @AmeliaNadal
grid	egon_etrago_generator	YES	@ClaraBuettner @AmeliaNadal
grid	egon_etrago_generator_timeseries	YES	@ClaraBuettner @AmeliaNadal
grid	egon_etrago_line	YES	@ClaraBuettner
grid	egon_etrago_line_timeseries	YES	@ClaraBuettner
grid	egon_etrago_link	YES	@ClaraBuettner @AmeliaNadal
grid	egon_etrago_link_timeseries	YES	@ClaraBuettner @AmeliaNadal
grid	egon_etrago_load	YES	@ClaraBuettner @AmeliaNadal
grid	egon_etrago_load_timeseries	YES	@ClaraBuettner @AmeliaNadal
grid	egon_etrago_storage	YES	@ClaraBuettner @AmeliaNadal
grid	egon_etrago_storage_timeseries	YES	@ClaraBuettner @AmeliaNadal
grid	egon_etrago_store	YES	@ClaraBuettner @AmeliaNadal
grid	egon_etrago_store_timeseries	YES	@ClaraBuettner @AmeliaNadal
grid	egon_etrago_temp_resolution	YES	@ClaraBuettner @AmeliaNadal
grid	egon_etrago_transformer	YES	@ClaraBuettner @AmeliaNadal
grid	egon_etrago_transformer_timeseries	YES	@ClaraBuettner @AmeliaNadal
grid	egon_voronoi_ch4	NO
grid	egon_voronoi_h2	NO
grid	egon_hvmv_substation	YES	@IlkaCu
grid	egon_hvmv_substation_voronoi	NO
grid	egon_mv_grid_district (check PR523)	YES	@nesnoj
grid	osm_point
grid	osm_polygon
grid	osm_roads
grid	destatis_zensus_population_per_ha_inside_germany
grid	egon_destatis_zensus_building_per_ha
grid	vg250_gem_population
grid	egon_destatis_zensus_apartment_per_ha
grid	egon_building_electricity_peak_loads
grid	egon_destatis_zensus_apartment_building_population_per_ha
grid	egon_destatis_zensus_household_per_ha
grid	egon_population_prognosis
grid	egon_household_prognosis
grid	egon_district_heating_areas
grid	egon_map_zensus_district_heating_areas
grid	egon_ehv_transfer_buses
grid	osm_landuse
grid	egon_hvmv_transfer_buses
grid	egon_loadarea
grid	osm_amenities_shops_filtered
grid	egon_etrago_timeseries_individual_heating
grid	branch_data
grid	dcline_data
grid	osm_buildings_residential
grid	osm_amenities_in_buildings_filtered
grid	osm_buildings_filtered_with_amenities
grid	osm_amenities_not_in_buildings_filtered
grid	osm_amenities_not_in_buildings_filtered
metadata	datasets	NO
metadata	dependency_graph	NO
openstreetmap	osm_amenities_in_buildings_tmp	NO
openstreetmap	osm_amenities_not_in_buildings	YES	@nesnoj
openstreetmap	osm_amenities_shops_filtered	YES	@nesnoj
openstreetmap	osm_buildings	YES	@nesnoj
openstreetmap	osm_buildings_filtered	YES	@nesnoj
openstreetmap	osm_buildings_with_amenities	YES	@nesnoj
openstreetmap	osm_buildings_with_res_tmp	NO
openstreetmap	osm_buildings_with_res_tmp2	NO
openstreetmap	osm_landuse	YES	@IlkaCu
openstreetmap	osm_line	YES	@nesnoj	X
openstreetmap	osm_nodes	YES	@nesnoj	X
openstreetmap	osm_point	YES	@nesnoj	X
openstreetmap	osm_polygon	YES	@nesnoj	X
openstreetmap	osm_rels	YES	@nesnoj	X
openstreetmap	osm_roads	YES	@nesnoj	X
openstreetmap	osm_ways	YES	@nesnoj	X
openstreetmap	osm_ways_preprocessed	YES	@nesnoj
openstreetmap	osm_ways_with_segments	YES	@nesnoj
openstreetmap	egon_building_heat_peak_loads
openstreetmap	egon_ehv_substation
openstreetmap	egon_ehv_substation_voronoi
openstreetmap	egon_scenario_capacities
openstreetmap	egon_nep_2021_conventional_powerplants
openstreetmap	egon_mv_grid_district
openstreetmap	egon_map_mvgriddistrict_vg250
openstreetmap	iee_household_load_profiles
openstreetmap	egon_destatis_zensus_household_per_ha_refined
openstreetmap	egon_household_electricity_profile_in_census_cell
openstreetmap	osm_buildings_synthetic	YES
openstreetmap	egon_household_electricity_profile_of_buildings
openstreetmap	egon_saltstructures_storage_potential
openstreetmap	egon_etrago_electricity_households
openstreetmap	egon_demandregio_sites_ind_electricity
openstreetmap	egon_demandregio_osm_ind_electricity
openstreetmap	egon_osm_ind_load_curves
openstreetmap	egon_osm_ind_load_curves_individual
osmtgmod_results	branch_data	NO
osmtgmod_results	bus_data	NO
osmtgmod_results	dcline_data	NO
osmtgmod_results	results_metadata	NO
osmtgmod_results	egon_sites_ind_load_curves
osmtgmod_results	egon_sites_ind_load_curves_individual
osmtgmod_results	egon_demandregio_zensus_electricity
public	_db_status	NO
public	abstr_values	NO
public	branch_data	NO
public	branch_specifications	NO
public	bus_data	NO
public	dcline_data	NO
public	dcline_specifications	NO
public	debug	NO
public	edit_power_relations	NO
public	nodes	NO
public	nuts_poly	NO
public	osm_metadata	NO
public	plz_poly	NO
public	power_circ_members	NO
public	power_circuits	NO
public	power_line	NO
public	power_line_sep	NO
public	power_relations	NO
public	power_substation	NO
public	power_ways	NO
public	problem_log	NO
public	relation_members	NO
public	relations	NO
public	schema_info	NO
public	spatial_ref_sys	NO
public	transfer_busses	NO
public	transfer_busses_complete	NO
public	transfer_busses_connect	NO
public	transfer_busses_connect_all	NO
public	transformer_specifications	NO
public	users	NO
public	way_nodes	NO
public	ways	NO
scenario	egon_scenario_parameters	YES
scenario	egon_etrago_electricity_cts_dsm_timeseries
society	destatis_zensus_population_per_ha	YES	@IlkaCu
society	destatis_zensus_population_per_ha_inside_germany	YES	@nesnoj	X
society	egon_demandregio_household	NO
society	egon_demandregio_population	NO
society	egon_destatis_zensus_apartment_building_population_per_ha	YES	@nesnoj
society	egon_destatis_zensus_apartment_per_ha	YES	@IlkaCu
society	egon_destatis_zensus_building_per_ha	YES	@IlkaCu
society	egon_destatis_zensus_household_per_ha	YES	@IlkaCu
society	egon_household_prognosis	YES	@IlkaCu
society	egon_population_prognosis	YES	@IlkaCu
society	egon_osm_ind_load_curves_individual_dsm_timeseries
society	egon_etrago_ac_h2
society	egon_etrago_ch4_h2
society	egon_gas_voronoi
society	egon_emob_charging_infrastructure	YES
society	egon_heavy_duty_transport_voronoi
society	egon_chp_plants	YES
society	egon_mastr_conventional_without_chp
supply	egon_chp	YES	@ClaraBuettner
supply	egon_district_heating	YES	@ClaraBuettner
supply	egon_era5_renewable_feedin	YES	@ClaraBuettner
supply	egon_era5_weather_cells	YES	@ClaraBuettner
supply	egon_individual_heating	YES	@ClaraBuettner
supply	egon_mastr_conventional_without_chp	NO
supply	egon_power_plants	YES	@IlkaCu
supply	egon_re_potential_area_pv_agriculture	YES	@nesnoj
supply	egon_re_potential_area_pv_road_railway	YES	@nesnoj
supply	egon_re_potential_area_wind	YES	@nesnoj
supply	egon_scenario_capacities	YES	@ClaraBuettner
supply	nep_2021_conv_powerplants	NO
supply	egon_mastr_pv_roof_geocoded
supply	egon_power_plants_pv_roof_building	YES
supply	egon_storages
supply	egon_home_batteries	YES
supply	egon_power_plants_wind	YES
supply	egon_power_plants_pv	YES
supply	egon_power_plants_biomass	YES
supply	egon_power_plants_hydro	YES

nesnoj commented 3 years ago

Alright, now I'm all done with the metadata instructions and examples. Let's do some meta :)

nesnoj commented 1 year ago

We created some metadata 4 the HH paper which could be implemented

ClaraBuettner commented 1 year ago

Since the table in the issue description is not that easy to edit and read, I created one on the cloud (1_Arbeitsinhalte/AP1/metadata/). I extracted the list of tables from one of the last CI-runs, so it should be up to date. We will take care about moving all the entries from the table in this issue to the table. But not all rows are filled here, so we might need some help from you.

birgits commented 1 year ago

Thank you @ClaraBuettner :) Will you let us know, when entries are moved to the cloud table? Then we will continue filling it out there.

nesnoj commented 1 year ago

Some thoughts about table relations: I think it'd be helpful to know to what other table the fields in resources fields refer to. A good place could be the field description. Something like

            "schema": {
                "fields": [
                    {
                        "name": "zensus_population_id",
                        "description": "ID of Census cell, refers to field society.destatis_zensus_population_per_ha_inside_germany.id",
                        "type": "serial",
                        "unit": null
                    },

Or, if applicable, via the foreign key. This makes it easier for users to join multiple tables (additional to example queries).

ClaraBuettner commented 1 year ago

Thank you @ClaraBuettner :) Will you let us know, when entries are moved to the cloud table? Then we will continue filling it out there.

Yes, @Alonsoju95 will let you know.

Alonsoju95 commented 1 year ago

Hi, everyone! All entries have been moved to the cloud table. The link: https://wolke.rl-institut.de/s/9RyJKLP2HRGxdYf

IlkaCu commented 1 year ago

I created a new tab 'Tables' in the mentioned document on the nextcloud, where I removed all the duplicates etc.

khelfen commented 1 year ago

For the following tables I need the sources even if they are not in need of metadata:

demand.egon_etrago_electricity_cts demand.egon_demandregio_sites_ind_electricity demand.egon_osm_ind_load_curves_individual demand.egon_sites_ind_load_curves_individual

openego / eGon-data

Create metadata for existing datasets #450

List of DB tables