Revisit Clustering Weighting per Balancing Area

Checklist

[X] I am using the current master branch
[X] I am running on an up-to-date pypsa-usa environment. Update via conda env update -f envs/environment.yaml

The Issue

With a high renewable system, the method pypsa-eur weights each cluster begins with weighting the zones by load and conventional generators. When we use ADS configuration with larger number of nodes (300), there are many substations with little or none of these gens.

Revisit this clustering method.

Steps To Reproduce

scenarios:
  all:
    interconnect: western #"usa|texas|western|eastern"
    clusters: 300
    opts:
      [
        Co2L0.75,
      ]
    ll: [vopt]
    scope: "total" # "urban", "rural", or "total" 

run:
  name: "ads_osw16_noUpgrade" # use this to keep track of runs with different settings
  disable_progressbar: false # set to true to disable the progressbar
  shared_resources: false # set to true to share the default resources across runs
  shared_cutouts: true # set to true to share the default cutout(s) across runs

enable:
  build_cutout: false

offshore_shape:
  use: ca #options are ca, weathergov
  offshore_path:
    ca: "repo_data/BOEM_CA_OSW_GIS/CA_OSW_BOEM_CallAreas.shp"
    weathergov: "https://www.weather.gov/source/gis/Shapefiles/WSOM/oz22mr22.zip"
    # nrel: "geodata_repo/Offshore_Wind_Speed_90m/Offshore_Wind_Speed_90m.shp"

balancing_area_gis: 
  path: "repo_data/BA_shapes_new/Modified_BE_BA_Shapes.shp"

#TODO: #19 remove config list of BA's and reference geojson values directly instead
countries: [US]  

# focus_weights: 
#   CISO-PGAE: 0.2

# TODO: #11 change load data and generator data to be a snakemake wildcard value
network_configuration: "ads2032" # "pypsa-usa" or "ads2032" or "breakthrough"

snapshots:
  start: "2019-01-01"
  end: "2019-12-31"
  inclusive: 'left' # include start, not end

atlite:
  default_cutout: era5_2019
  nprocesses: 4
  show_progress: false # false saves time
  cutouts:
    era5_2019:
      module: era5 # in priority order
      time: ['2019', '2019']
  interconnects:
    western:
      x: [-126, -99]
      y: [27, 50]
      dx: 0.3
      dy: 0.3
    eastern:
      x: [-109, -65]
      y: [23, 50]
      dx: 0.3
      dy: 0.3
    texas:
      x: [-110,-90]
      y: [24, 37]
      dx: 0.3
      dy: 0.3
    usa:
      x: [-126, -65]
      y: [23, 50]
      dx: 0.3
      dy: 0.3

electricity: 
  conventional_carriers: [nuclear, oil, OCGT, CCGT, coal, geothermal] # Choose the conventional plant types to include in network
  renewable_carriers: [onwind, solar, offwind, hydro] # Choose the renewable plant types to include in network
  voltages: [230., 345., 500., 765.] #remove
  voltage_simplified: 230 #Voltage level to simplify network to in rule "simplify network"
  co2limit: 1.4728e+9 # 0.8 * 1.841e+9
  co2base: 1.841e+9 #base_from_2020 Locations of the 1,841 MMmt of CO2 emissions from the current state of the grid in 2020.
  gaslimit: false # global gas usage limit of X MWh_th

  max_hours:
    battery: 6 # ADS USES THIS AS ASSUMPTION FOR CURRENT BATTERIES
    H2: 168

  extendable_carriers:
    Generator: [solar, onwind, offwind, OCGT, CCGT, coal]
    StorageUnit: [battery] # battery, H2
    Store: [battery] #[battery, H2]
    Link: [] #[H2 pipeline]

conventional:
  unit_commitment: false
  dynamic_fuel_price: false

renewable: #build_renewable_profiles configurations
  onwind:
    cutout: era5_2019
    resource:
      method: wind
      turbine: Vestas_V112_3MW
    capacity_per_sqkm: 3 # conservative, ScholzPhd Tab 4.3.1: 10MW/km^2
    # correction_factor: 0.93
    corine:
      #all keys labeled corrine are actually copernicus codes. Using the name corrine bc using the pypsa-eur convention
      # Scholz, Y. (2012). Renewable energy based electricity supply at low costs:
      #  development of the REMix model and application for Europe. ( p.42 / p.28)
      # CLC grid codes:
      # 11X/12X - Various forest types
      # 20  - Shrubs
      # 30  - Herbaceus vegetation
      # 40  - Cropland
      # 50  - Urban
      # 60  - Bare / Sparse vegetation
      # 80  - Permanent water bodies
      # 100 - Moss and lichen
      # 200 - Open sea
      grid_codes: [20, 30, 40, 60, 100, 111, 112, 113, 114, 115, 116, 121, 122, 123, 124, 125, 126]
      distance: 10 #buffer from distance_grid_codes that are to be excluded
      distance_grid_codes: [50] 
    natura: true
    potential: simple # or conservative
    clip_p_max_pu: 1.e-2
    extendable: true
  offwind:
    cutout: era5_2019
    resource:
      method: wind
      turbine: NREL_ReferenceTurbine_5MW_offshore
    capacity_per_sqkm: 2
    correction_factor: 0.8855
    # proxy for wake losses
    # from 10.1016/j.energy.2018.08.153
    # until done more rigorously in #153
    corine:
      grid_codes: [80, 200] #page 28 of https://land.copernicus.eu/global/sites/cgls.vito.be/files/products/CGLOPS1_PUM_LC100m-V3_I3.4.pdf
    natura: true
    max_depth: 1000
    max_shore_distance: 30000
    potential: simple # or conservative
    clip_p_max_pu: 1.e-2
    extendable: true
  solar:
    cutout: era5_2019
    resource:
      method: pv
      panel: CSi
      orientation: latitude_optimal # will lead into optimal design
        # slope: 0.  # slope: 0 represent a flat panel
        # azimuth: 180.  # azimuth: 180 south orientation
    capacity_per_sqkm: 4.6 # From 1.7 to 4.6 addresses issue #361
    # Determined by comparing uncorrected area-weighted full-load hours to those
    # published in Supplementary Data to
    # Pietzcker, Robert Carl, et al. "Using the sun to decarbonize the power
    # sector: The economic potential of photovoltaics and concentrating solar
    # power." Applied Energy 135 (2014): 704-720.
    correction_factor: 0.854337
    corine:
      grid_codes: [20, 30, 40, 50, 60, 90, 100] #see above for codes
    natura: true
    potential: simple # or conservative
    clip_p_max_pu: 1.e-2
    extendable: true
  hydro:
    cutout: era5_2019
    resource:
      method: hydro
      hydrobasins: resources/hybas_na_lev06_v1c.shp
      flowspeed: 1.0  # m/s
      # weight_with_height: false
      # show_progress: true
    carriers: [ror, PHS, hydro]
    PHS_max_hours: 6
    hydro_max_hours: "energy_capacity_totals_by_country"  # not active
    clip_min_inflow: 1.0
    extendable: true
    normalization:
      method: hydro_capacities  # 'hydro_capacities' to rescale country hydro production by using hydro_capacities, 'eia' to rescale by eia data, false for no rescaling
      year: 2013  # (optional) year of statistics used to rescale the runoff time series. When not provided, the weather year of the snapshots is used
    multiplier: 1.1  # multiplier applied after the normalization of the hydro production; default 1.0

lines:
  types: # All temporary values, need to be updated
    115.: "Al/St 240/40 2-bundle 220.0" 
    138.: "Al/St 240/40 2-bundle 220.0"
    161.: "Al/St 240/40 2-bundle 220.0"
    230.: "Al/St 240/40 2-bundle 220.0"
    345.: "Al/St 240/40 4-bundle 380.0"
    500.: "Al/St 560/50 4-bundle 750.0"
    765.: "Al/St 560/50 4-bundle 750.0"
  s_max_pu: 0.7
  s_nom_max: .inf
  length_factor: 1.25
  under_construction: 'zero' # 'zero': set capacity to zero, 'remove': remove, 'keep': with full capacity

links:
  p_max_pu: 1.0
  p_nom_max: .inf
  include_tyndp: true
  under_construction: 'zero' # 'zero': set capacity to zero, 'remove': remove, 'keep': with full capacity

load:
  # power_statistics: true # only for files from <2019; set false in order to get ENTSOE transparency data
  # interpolate_limit: 3 # data gaps up until this size are interpolated linearly
  # time_shift_for_large_gaps: 1w # data gaps up until this size are copied by copying from
  # manual_adjustments: true # false
  scaling_factor: 1.0

costs:  # based on the potentials, assuming  (0.1 kW/m2 and 10 m2/person)
  year: 2030 
  version: v0.6.0
  rooftop_share: 0.14
  fill_values:
    FOM: 0
    VOM: 0
    efficiency: 1
    fuel: 0
    investment: 0
    lifetime: 25
    "CO2 intensity": 0
    "discount rate": 0.07
  marginal_cost:
    solar: 0.00
    onwind: 0.00
    offwind: 0.00
    hydro: 0.
    H2: 0.
    electrolysis: 0.
    fuel cell: 0.
    battery: 0.
    battery inverter: 0.
  emission_prices: # in currency per tonne emission, only used with the option Ep
    co2: 0.

sector:
  heat_pump_sink_T: 55.

clustering:
  simplify_network:
    to_substations: false # network is simplified to nodes with positive or negative power injection (i.e. substations or offwind connections)
    algorithm: kmeans # choose from: [hac, kmeans]
    feature: solar+onwind-time # only for hac. choose from: [solar+onwind-time, solar+onwind-cap, solar-time, solar-cap, solar+offwind-cap] etc.
  cluster_network:
    algorithm: kmeans
    feature: solar+onwind-time
    aggregation_zones: 'balancing_area' # balancing_area, country, or state. # Currently issue in State aggregation- error thrown on Utah
  aggregation_strategies:
    generators:
      p_nom_max: sum # use "min" for more conservative assumptions
      p_nom_min: sum
      p_min_pu: mean
      marginal_cost: mean
      committable: any
      ramp_limit_up: max
      ramp_limit_down: max
      efficiency: mean
    buses:
      Pd: sum
      zone_id: mean # should delete zone id's and PDs from network earlier.
      state: max

solving:
  options:
    operations_only: true
    formulation: kirchhoff
    load_shedding: true
    noisy_costs: true
    min_iterations: 4
    max_iterations: 6
    clip_p_max_pu: 0.01
    skip_iterations: true
    track_iterations: false
    nhours: 8760
  solver:
    name: gurobi
    threads: 4
    method: 2 # barrier
    crossover: 0
    BarConvTol: 1.e-5
    FeasibilityTol: 1.e-6
    AggFill: 0
    PreDual: 0
    GURO_PAR_BARDENSETHRESH: 200

plotting:
  map:
    figsize: [7, 7]
    boundaries: [-10.2, 29, 35,  72]
    p_nom:
      bus_size_factor: 5.e+4
      linewidth_factor: 3.e+3

  costs_max: 800
  costs_threshold: 1

  energy_max: 15000.
  energy_min: -10000.
  energy_threshold: 50.

  vre_techs: ["onwind", "offwind-ac", "offwind-dc", "solar", "ror"]
  conv_techs: ["OCGT", "CCGT", "Nuclear", "Coal"]
  storage_techs: ["hydro+PHS", "battery", "H2"]
  load_carriers: ["AC load"]
  AC_carriers: ["AC line", "AC transformer"]
  link_carriers: ["DC line", "Converter AC-DC"]
  tech_colors:
    "onwind" : "#235ebc"
    "wind" : "#235ebc"
    "onshore wind" : "#235ebc"
    'offwind' : "#dd6895"
    'offwind-ac' : "#6895dd"
    'offshore wind' : "#6895dd"
    'offshore wind ac' : "#6895dd"
    'offwind-dc' : "#74c6f2"
    'offshore wind dc' : "#74c6f2"
    "hydro" : "#08ad97"
    "hydro+PHS" : "#08ad97"
    "PHS" : "#08ad97"
    "hydro reservoir" : "#08ad97"
    'hydroelectricity' : '#08ad97'
    "ror" : "#4adbc8"
    "run of river" : "#4adbc8"
    'solar' : "#f9d002"
    'solar PV' : "#f9d002"
    'solar thermal' : '#ffef60'
    'biomass' : '#0c6013'
    'solid biomass' : '#06540d'
    'biogas' : '#23932d'
    'waste' : '#68896b'
    'geothermal' : '#ba91b1'
    "OCGT" : "#d35050"
    "gas" : "#d35050"
    "ng" : "#d35050"
    "natural gas" : "#d35050"
    "CCGT" : "#b20101"
    "nuclear" : "#ff9000"
    "coal" : "#707070"
    "lignite" : "#9e5a01"
    "oil" : "#262626"
    "H2" : "#ea048a"
    "hydrogen storage" : "#ea048a"
    "battery" : "#b8ea04"
    "Electric load" : "#f9d002"
    "electricity" : "#f9d002"
    "lines" : "#70af1d"
    "transmission lines" : "#70af1d"
    "AC-AC" : "#70af1d"
    "AC line" : "#70af1d"
    "AC" : "#70af1d"
    "links" : "#8a1caf"
    "HVDC links" : "#8a1caf"
    "DC-DC" : "#8a1caf"
    "DC link" : "#8a1caf"
    "DC" : "#8a1caf"

  nice_names:
    OCGT: "Open-Cycle Gas"
    CCGT: "Combined-Cycle Gas"
    offwind-ac: "Offshore Wind (AC)"
    offwind-dc: "Offshore Wind (DC)"
    onwind: "Onshore Wind"
    solar: "Solar"
    PHS: "Pumped Hydro Storage"
    hydro: "Reservoir & Dam"
    battery: "Battery Storage"
    H2: "Hydrogen Storage"
    lines: "Transmission Lines"
    ror: "Run of River"

Expected Behavior

No response

Error Message

clustering.py directly
INFO:pypsa.io:Imported network elec_s.nc has buses, carriers, generators, lines, links, loads, storage_units
WARNING:__main__:Keyword argument feature is only valid for algorithm `hac`. Given feature `solar+onwind-time` will be ignored.
Traceback (most recent call last):
  File "/Users/kamrantehranchi/Local_Documents/pypsa-usa/workflow/.snakemake/scripts/tmpjy7_wenh.cluster_network_eur.py", line 420, in <module>
    clustering = clustering_for_n_clusters(n, n_clusters, custom_busmap, aggregate_carriers,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kamrantehranchi/Local_Documents/pypsa-usa/workflow/.snakemake/scripts/tmpjy7_wenh.cluster_network_eur.py", line 313, in clustering_for_n_clusters
    busmap = busmap_for_n_clusters(n, n_clusters, solver_name, focus_weights, algorithm, feature)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kamrantehranchi/Local_Documents/pypsa-usa/workflow/.snakemake/scripts/tmpjy7_wenh.cluster_network_eur.py", line 302, in busmap_for_n_clusters
    .apply(busmap_for_country).squeeze().rename('busmap'))
     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kamrantehranchi/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/groupby/groupby.py", line 1353, in apply
    result = self._python_apply_general(f, self._selected_obj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kamrantehranchi/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/groupby/groupby.py", line 1402, in _python_apply_general
    values, mutated = self.grouper.apply(f, data, self.axis)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kamrantehranchi/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/groupby/ops.py", line 767, in apply
    res = f(group)
          ^^^^^^^^
  File "/Users/kamrantehranchi/Local_Documents/pypsa-usa/workflow/.snakemake/scripts/tmpjy7_wenh.cluster_network_eur.py", line 290, in busmap_for_country
    weight = weighting_for_country(n, x)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kamrantehranchi/Local_Documents/pypsa-usa/workflow/.snakemake/scripts/tmpjy7_wenh.cluster_network_eur.py", line 151, in weighting_for_country
    return (w * (100. / w.max())).clip(lower=1.).astype(int)
                 ~~~~~^~~~~~~~~
ZeroDivisionError: float division by zero

Anything else?