search

calliope-project / euro-calliope

A workflow to build models of the European electricity system for Calliope.
https://euro-calliope.readthedocs.io
MIT License
31 stars 18 forks source link

Transport demand workflow missing country data infilling #395

Open brynpickering opened 5 months ago

brynpickering commented 5 months ago

What happened?

Running the full workflow, it has become apparent that the rule create_controlled_road_transport_annual_demand_and_installed_capacities misses data from some countries. This creates an error in the workflow in this set of method calls: https://github.com/calliope-project/euro-calliope/blob/5d5b4eb2b54c9381c569a2230e302e9014173a00/scripts/transport/road_transport_controlled_charging.py#L95-L104

Version

1.2.0.dev

Relevant log output


Traceback (most recent call last):
  File "/Users/bryn.pickering/Repos/calliope-project/euro-calliope/.snakemake/scripts/tmpwia8kyhz.road_transport_controlled_charging.py", line 159, in <module>
    df_charging_potentials = extract_national_ev_charging_potentials(
  File "/Users/bryn.pickering/Repos/calliope-project/euro-calliope/.snakemake/scripts/tmpwia8kyhz.road_transport_controlled_charging.py", line 100, in extract_national_ev_charging_potentials
    df_ev_numbers.align(battery_size, level="vehicle_type")[1]
  File "/Users/bryn.pickering/Repos/calliope-project/envs/snakemake/euro-calliope/ca1e86f4514e6b35aa58fd05115d87c1_/lib/python3.9/site-packages/pandas/core/indexing.py", line 889, in __getitem__
    return self._getitem_tuple(key)
  File "/Users/bryn.pickering/Repos/calliope-project/envs/snakemake/euro-calliope/ca1e86f4514e6b35aa58fd05115d87c1_/lib/python3.9/site-packages/pandas/core/indexing.py", line 1060, in _getitem_tuple
    return self._getitem_lowerdim(tup)
  File "/Users/bryn.pickering/Repos/calliope-project/envs/snakemake/euro-calliope/ca1e86f4514e6b35aa58fd05115d87c1_/lib/python3.9/site-packages/pandas/core/indexing.py", line 791, in _getitem_lowerdim
    return self._getitem_nested_tuple(tup)
  File "/Users/bryn.pickering/Repos/calliope-project/envs/snakemake/euro-calliope/ca1e86f4514e6b35aa58fd05115d87c1_/lib/python3.9/site-packages/pandas/core/indexing.py", line 865, in _getitem_nested_tuple
    obj = getattr(obj, self.name)._getitem_axis(key, axis=axis)
  File "/Users/bryn.pickering/Repos/calliope-project/envs/snakemake/euro-calliope/ca1e86f4514e6b35aa58fd05115d87c1_/lib/python3.9/site-packages/pandas/core/indexing.py", line 1113, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
  File "/Users/bryn.pickering/Repos/calliope-project/envs/snakemake/euro-calliope/ca1e86f4514e6b35aa58fd05115d87c1_/lib/python3.9/site-packages/pandas/core/indexing.py", line 1053, in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis, raise_missing=False)
  File "/Users/bryn.pickering/Repos/calliope-project/envs/snakemake/euro-calliope/ca1e86f4514e6b35aa58fd05115d87c1_/lib/python3.9/site-packages/pandas/core/indexing.py", line 1254, in _get_listlike_indexer
    indexer, keyarr = ax._convert_listlike_indexer(key)
  File "/Users/bryn.pickering/Repos/calliope-project/envs/snakemake/euro-calliope/ca1e86f4514e6b35aa58fd05115d87c1_/lib/python3.9/site-packages/pandas/core/indexes/multi.py", line 2568, in _convert_listlike_indexer
    raise KeyError(f"{keyarr[mask]} not in index")
KeyError: "['ALB' 'BIH' 'MKD' 'MNE' 'NOR' 'SRB' 'CHE'] not in index"
adrienmellot commented 5 months ago

Good spot, I guess that's the issue of usually only running the minimal workflow. Perhaps we should adjust the minimal workflow to include one of the countries for which data is usually missing? It seems that it's the JRC data missing countries. Should we directly address that in the JRC data processing upstream or should we do it in the transports script directly?

adrienmellot commented 5 months ago

Addressed in #394 directly as the code is anyway modified.