Add the option to define, create and remove subregions

[virio-andreyana]

Closes https://github.com/pypsa-meets-earth/pypsa-earth/issues/788 (the simulate subregion part)

subregion:
  define_subregion_gadm:
  drop_subregion: 

• define_subregion_gadm: Chose the subregion based on its GADM subdivision Map ID code.
• Chose the country in https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg and use https://ngageoint.github.io/geopackage-viewer-js/ and chose ADM_ADM_1 to identify the GADM ID.
• This will take away the land of the GADM ID code from the original country to the new country.
• One subregion can be composed of multiple countries.
• drop_subregion: remove the country/subregion from the model.

When creating subregions, its loads and existing renewable capacity will be empty. Hence it is recommended to use this additional tools as well

load_options:
  scale:
  subtitute_country_load:

estimate_renewable_capacities:
  custom_capacity:
    Offshore:
    Onshore:
    PV:

Here is an example of countries divided into subregions:

subregion:
  define_subregion_gadm:
    Java-Bali/ID:
      ID: [2,4,7,9,10,11,33]
    Peninsular/MY:
      MY: [1,2,3,4,6,7,8,9,10,11,12,15,16]
    Papua/ID:
      ID: [22,23]
    Sarawak/MY:
      MY: [14]
    Maluku/ID:
      ID: [18,19]
    Sulawesi/ID:
      ID: [6,25,26,27,28,29]
    Sabah/MY:
      MY: [5,13]
    Kalimantan/ID:
      ID: [12,13,14,34,35]
    Nusa-Tenggara/ID:
      ID: [20,21]
    Sumatra/ID:
      ID: [1,3,5,8,16,17,24,30,31,32]

  drop_subregion: ["ID","MY"]

Changes proposed in this Pull Request

in snakemake

• a new config called "country_plus" is made which is the list of countries and subregion (also removed the dropped countries/subregion)
• this config "country_plus" will play an impact in
  • build_osm_network
  • base_network
  • build_bus_regions
  • build_renewable_profiles
  • build_powerplants (uses both countries and countries_plus)
  • simplify_network
  • cluster_network
  • all the summaries rules (make_all_summaries, plot_all_summaries)

in build_shapes:

• this is the main method of how the new subregion is defined
• buses and transmission line follow suit using these new definition.

in add_electricity:

• custom_capacity is added for the IRENA list because subregion are not in IRENA list.
• to get a result similar to an undivided country, you first have to run the full country and see the capacity distribution of each GADM region you include.

in build_demand_profiles:

• subtitute_country_load is added because specific countries (ex. Laos) and subregion are not in GEIGIS list.
• to get a result similar to an undivided country, you first have to run the full country and see the load distribution of each GADM region you include.

in build_powerplants:

• the list of powerplants do not initially recognize the new subregion.
• this helps the powerplant to refer to the buses in the new subregion.

in simplify_network:

• a minor bug is fix. Using distance_crs instead of geo_crs because sjoin_nearest requires accurated distance.
• add the option to skip certain countries from merging (very specific for island countries)

in test/config.custom.yaml

• run the simulation for creating subregion to make sure that its working.

note:

• it is unclear if using country_plus has a negative impact on build_renewable_profiles. They need to extract IEA data in the csv file, but inspite of that, the subregion do include the hydro capacity as well.

Checklist

• [x] I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• [ ] I tested my contribution locally and it seems to work fine.
• [ ] Code and workflow changes are sufficiently documented.
• [ ] Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• [ ] Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• [ ] Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• [ ] Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• [ ] A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[davide-f]

The approach is very interesting and it accounts for the problem of the demand rescaling that the previous formulation had. This PR not only addresses the issue of the subregions but increases significantly the flexibility in defining the regions. The approach is interesting, though I'm wondering if there may be easier approaches to do the same. An approach crossed my mind is to revise the use of the alternative_clustering and:

1. disentangle the use of custom shapes in build_bus_regions from the ones used in cluster_network. This may require a little revision on the gadm_shapes parameter of the two approaches, which in principle can be any shape, not necessarily the gadm one
2. allow the automatic creation of a custom clustering process in the cluster_network phase that drops/merges the subregions of the gadm_shapes given the desired interests.

What do you think? We can mention it also with the others during the meeting.

Multiple opinions by other developers are highly desiderable too

[virio-andreyana]

Interesting @davide-f! Most of the challange this draft made really happens in the process of turning base.nc into elec.nc (load adjustment, renewables, powerplant) because they specifically need the 2_country_code. Finding the correct settings for all of the subregion is more of a hassle that I would imagine people like to have

However, I would like for the subregion to be defined before elec_s.nc because thats where the backbone algorithm takes place. Defining the subregion starting from cluster_network.py phase would cause back my original issue.

A good compromise maybe is the way similar to how augmented_line_connections.py is conducted. Have a snakefile if function where if subregions are defined, all the buses, lines, and generators within that region countries are renamed. This can take place after elec.nc but before elec_s.nc. That way, you have the advantage of both worlds.

My main concern is the way in which the renewables in offshore regions are impacted. The current method also cuts the eez zone into subregions as well. How accurate is the voronoi in build_renewables.py so that it didn't go too far or too short compared to the current results.

That's my 2 cents. I'll see you in the meeting.

[davide-f]

However, I would like for the subregion to be defined before elec_s.nc because thats where the backbone algorithm takes place. Defining the subregion starting from cluster_network.py phase would cause back my original issue.

Could you remind what are the problems in the simplifying you were experiencing?

A good compromise maybe is the way similar to how augmented_line_connections.py is conducted. Have a snakefile if function where if subregions are defined, all the buses, lines, and generators within that region countries are renamed. This can take place after elec.nc but before elec_s.nc. That way, you have the advantage of both worlds.

My main concern is the way in which the renewables in offshore regions are impacted. The current method also cuts the eez zone into subregions as well. How accurate is the voronoi in build_renewables.py so that it didn't go too far or too short compared to the current results.

Mmm I need to better explore this; we are aware of problems in the eez, but I didn't get well how this PR relates to that unfortunately