jnnr
commented
3 months ago This might be useful: The commit contains functions that map from EUROSTAT statistical data to NUTS units. https://github.com/calliope-project/euro-calliope/commit/e995c25318f7d20e31345b925cfc15638fbb473b#diff-94359c59da328aa237ab2d75320a84c6343fe4954a4169910fd9e0d376dc42eb
Plan
The goal is this: The workflow preprocesses all raw spatial data that is to be used later at sub-national resolution into clean raster data. This preprocessed data can then be aggregated in the next step to any custom resolution (e.g. regional, E-highways, some custom resolution that is high-res for one country and more coarse for another).
- Formulate general operations that are common. The operation performed is quite similar in all these places, so we will first define a common API and some functions that help doing the mapping. See #324
- We want to be able to rasterize statistical data that is defined on polygons:
rasterize_polygon_based_data(data, spatial_units, resolution_raster, proxy) - Point data will not be rasterized.
- We want to be able to rasterize statistical data that is defined on polygons:
- Decide on a common raster data format: GEOTiff or netCDF, which will be processed by the xarray library. Maybe not even necessary because we may just keep the format of the raw input data
- Focus on the heat sector for the start.: Start by disaggregating commercial heat demand by GVA in EC (scripts/heat/rescale.py)
- Additionally/alternatively, focus on rule potentials? Not necessary in the end, cause we can build ehighways (which is the short-term goal) already with existing zenodo data, as described in #319
- Later, create a list of all steps in the workflow where data is preprocessed at sub-national resolution.
First, understand how spatial units and data are currently processed
How does EC process spatial units and data?
Where does preparation of units happen? NUTS and GADM are downloaded and prepared.
unitscollects NUTS and GADM data, "Remixes NUTS, LAU, and GADM data to form the units of the analysis.", and provides input ("build/data/{resolution}/units.geojson") to many rules, namelyunits_without_shape(producing units.csv at a given resolution), used inaggregate_timeseries(electrified road transport). Sums timeseries for continental and national resolution, disaggregates to regions according to population for regional resolution.aggregate_timeseries_historic_electrified: Copy of above rule, but with historic electrification.biofuelsDisaggregates biofuel potential to regions using proxies, if elsearea_to_capacity_limitsuse unit names to reindex capacity data.nuclear_regional_capacity: geopandas.overlay operationelectricity_load: uses national demand and scales it to regions using predefined scaling factors in https://zenodo.org/record/5112963/files/possibility-for-electricity-autarky.ziplocations_template: Takes the regions id and adds the centroids as coordinates to the templatecapacity_factors_onshore_wind_and_solar: uses units to calculate area weighted timeserieshydro_capacitiesgeopandas.sjoincapacity_factors_hydrogeopandas.sjoinHow does SCEC process spatial units and data?
annual_subnational_demandregional_dwelling_ratiodata-sources.statistical_units_to_custom_regionscustom_units, which only exists on Bryn's fork. This rule creates "build/data/ehighways/units.geojson"copy_euro_calliopecopy_resolution_specific_euro_calliopeannual_waste_supplyannual_subnational_demandregionsweather_and_populationupdate_electricity_with_other_sectorsgas_storagecopy_biofuel_techscoal_supply_yamlTentative plan:
Set a NUTS year convention After data cleaning, in every instance, disaggregate timeseries or scalars to NUTS3 Later, aggregate. Mapping from NUTS3 to Ehighways is easy (up to differences caused by different NUTS years)