search

grid-parity-exchange / Egret

Tools for building power systems optimization problems
Other
126 stars 50 forks source link

TypeError: slice indices must be integers or None or have an __index__ method after updating to pandas 1.5 #290

Closed lbianchi-lbl closed 1 year ago

lbianchi-lbl commented 1 year ago

pandas was recently (yesterday?) updated to version 1.5, which seems to have introduced changes that are not compatible with the Egret internals.

We detected this when running the IDAES test suite, which runs (a version of) the Prescient 5bus example, resulting in the following stack trace:

_____________________________________________________ ERROR at setup of Test5Bus.test_simulation_results _____________________________________________________

self = <idaes.tests.prescient.test_prescient.Test5Bus object at 0x7f585b5fb670>
prescient_options = {'compute_market_settlements': True, 'contingency_price_threshold': 100, 'data_path': '/home/ludo/lbl/idaes/idaes-pse/idaes/tests/prescient/5bus', 'deterministic_ruc_solver': 'cbc', ...}

    @pytest.fixture
    def run_simulator(self, prescient_options: PrescientOptions) -> None:
        prescient_simulator = pytest.importorskip(
            "prescient.simulator",
            reason="Prescient (optional dependency) not available",
        )

        sim = prescient_simulator.Prescient()
>       sim.simulate(**prescient_options)

idaes/tests/prescient/test_prescient.py:91: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/conda/envs/test-idaes-967/lib/python3.9/site-packages/prescient/simulator/prescient.py:75: in simulate
    return self._simulate(self.config)
/opt/conda/envs/test-idaes-967/lib/python3.9/site-packages/prescient/simulator/prescient.py:82: in _simulate
    return super().simulate(options)
/opt/conda/envs/test-idaes-967/lib/python3.9/site-packages/prescient/simulator/simulator.py:95: in simulate
    oracle_manager.call_initialization_oracle(options, first_time_step)
/opt/conda/envs/test-idaes-967/lib/python3.9/site-packages/prescient/simulator/oracle_manager.py:125: in call_initialization_oracle
    self._generate_pending_ruc(options, time_step.date, time_step.hour, None)
/opt/conda/envs/test-idaes-967/lib/python3.9/site-packages/prescient/simulator/oracle_manager.py:198: in _generate_pending_ruc
    simulation_actuals = self._formulate_actuals(options, uc_date, uc_hour)
/opt/conda/envs/test-idaes-967/lib/python3.9/site-packages/prescient/simulator/oracle_manager.py:181: in _formulate_actuals
    return self.engine.create_simulation_actuals(
/opt/conda/envs/test-idaes-967/lib/python3.9/site-packages/prescient/engine/egret/engine.py:125: in create_simulation_actuals
    return self._p.create_simulation_actuals(options, self._data_provider,
/opt/conda/envs/test-idaes-967/lib/python3.9/site-packages/prescient/engine/egret/egret_plugin.py:628: in create_simulation_actuals
    get_data_func(options, start_time, total_step_count, step_size_minutes, md)
/opt/conda/envs/test-idaes-967/lib/python3.9/site-packages/prescient/data/providers/gmlc_data_provider.py:183: in populate_with_actuals
    self._populate_with_forecastable_data('REAL_TIME', start_time, num_time_periods, time_period_length_minutes, model)
/opt/conda/envs/test-idaes-967/lib/python3.9/site-packages/prescient/data/providers/gmlc_data_provider.py:199: in _populate_with_forecastable_data
    copy_from = self._cache.generate_model(sim_type, start_time, end_time)
/opt/conda/envs/test-idaes-967/lib/python3.9/site-packages/egret/parsers/rts_gmlc/parsed_cache.py:67: in generate_model
    self.populate_skeleton_with_data(md, simulation_type, begin_time, end_time)
/opt/conda/envs/test-idaes-967/lib/python3.9/site-packages/egret/parsers/rts_gmlc/parsed_cache.py:94: in populate_skeleton_with_data
    self._process_timeseries_data(skeleton_dict, simulation_type, begin_time, end_time)
/opt/conda/envs/test-idaes-967/lib/python3.9/site-packages/egret/parsers/rts_gmlc/parsed_cache.py:173: in _process_timeseries_data
    self._process_generator_timeseries(md, begin_time, end_time, i)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <egret.parsers.rts_gmlc.parsed_cache.ParsedCache object at 0x7f57e853edc0>
md = {'elements': {'area': {'1': {}, '2': {}}, 'branch': {'branch_1_10': {'angle_diff_max': 90, 'angle_diff_min': -90, 'bra... ...}, ...}, ...}, 'system': {'baseMVA': 100.0, 'name': 'RTS-GMLC', 'reference_bus': 'bus4', 'reference_bus_angle': 0}}
begin_time = datetime.datetime(2020, 7, 10, 0, 0), end_time = datetime.datetime(2020, 7, 11, 11, 59, 59), df_index = 8

    def _process_generator_timeseries(self, md:dict, begin_time:datetime,
                                      end_time:datetime, df_index:int):
        df = self.timeseries_df
        i = df_index
        gen_name = df.iat[i, df.columns.get_loc('Object')]
        gen_dict = md['elements']['generator'][gen_name]
        param = df.iat[i, df.columns.get_loc('Parameter')]
>       data = df.iat[i, df.columns.get_loc('Series')][begin_time:end_time].to_list()
E       TypeError: slice indices must be integers or None or have an __index__ method

/opt/conda/envs/test-idaes-967/lib/python3.9/site-packages/egret/parsers/rts_gmlc/parsed_cache.py:186: TypeError

After running pytest --pdb, it looks like the contents of self.timeseries_df["Series"] are a numpy array:

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /opt/conda/envs/test-idaes-967/lib/python3.9/site-packages/egret/parsers/rts_gmlc/parsed_cache.py(186)_process_generator_timeseries()
-> data = df.iat[i, df.columns.get_loc('Series')][begin_time:end_time].to_list()
(Pdb) df.iat[i, df.columns.get_loc("Series")]
array([12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  ,
       12.7  , 12.7  , 12.7  , 12.7  , 12.783, 12.867, 12.95 , 13.033,
       13.117, 13.2  , 13.283, 13.367, 13.45 , 13.533, 13.617, 13.7  ,
       14.717, 15.733, 16.75 , 17.767, 18.783, 19.8  , 20.817, 21.833,
       22.85 , 23.867, 24.883, 25.9  , 25.983, 26.067, 26.15 , 26.233,
       26.317, 26.4  , 26.483, 26.567, 26.65 , 26.733, 26.817, 26.9  ,
       27.958, 29.017, 30.075, 31.133, 32.192, 33.25 , 34.308, 35.367,
       36.425, 37.483, 38.542, 39.6  , 39.525, 39.45 , 39.375, 39.3  ,
       39.225, 39.15 , 39.075, 39.   , 38.925, 38.85 , 38.775, 38.7  ,
       38.658, 38.617, 38.575, 38.533, 38.492, 38.45 , 38.408, 38.367,
       38.325, 38.283, 38.242, 38.2  , 38.317, 38.433, 38.55 , 38.667,
       38.783, 38.9  , 39.017, 39.133, 39.25 , 39.367, 39.483, 39.6  ,
       39.525, 39.45 , 39.375, 39.3  , 39.225, 39.15 , 39.075, 39.   ,
       38.925, 38.85 , 38.775, 38.7  , 38.742, 38.783, 38.825, 38.867,
       38.908, 38.95 , 38.992, 39.033, 39.075, 39.117, 39.158, 39.2  ,
       39.158, 39.117, 39.075, 39.033, 38.992, 38.95 , 38.908, 38.867,
       38.825, 38.783, 38.742, 38.7  , 38.617, 38.533, 38.45 , 38.367,
       38.283, 38.2  , 38.117, 38.033, 37.95 , 37.867, 37.783, 37.7  ,
       37.742, 37.783, 37.825, 37.867, 37.908, 37.95 , 37.992, 38.033,
       38.075, 38.117, 38.158, 38.2  , 38.242, 38.283, 38.325, 38.367,
       38.408, 38.45 , 38.492, 38.533, 38.575, 38.617, 38.658, 38.7  ,
       38.617, 38.533, 38.45 , 38.367, 38.283, 38.2  , 38.117, 38.033,
       37.95 , 37.867, 37.783, 37.7  , 37.742, 37.783, 37.825, 37.867,
       37.908, 37.95 , 37.992, 38.033, 38.075, 38.117, 38.158, 38.2  ,
       38.2  , 38.2  , 38.2  , 38.2  , 38.2  , 38.2  , 38.2  , 38.2  ,
       38.2  , 38.2  , 38.2  , 38.2  , 38.158, 38.117, 38.075, 38.033,
       37.992, 37.95 , 37.908, 37.867, 37.825, 37.783, 37.742, 37.7  ,
       37.7  , 37.7  , 37.7  , 37.7  , 37.7  , 37.7  , 37.7  , 37.7  ,
       37.7  , 37.7  , 37.7  , 37.7  , 36.683, 35.667, 34.65 , 33.633,
       32.617, 31.6  , 30.583, 29.567, 28.55 , 27.533, 26.517, 25.5  ,
       24.433, 23.367, 22.3  , 21.233, 20.167, 19.1  , 18.033, 16.967,
       15.9  , 14.833, 13.767, 12.7  , 12.7  , 12.7  , 12.7  , 12.7  ,
       12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  ,
       12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  ,
       12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  ,
       12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  ,
       12.667, 12.633, 12.6  , 12.567, 12.533, 12.5  , 12.467, 12.433,
       12.4  , 12.367, 12.333, 12.3  , 12.333, 12.367, 12.4  , 12.433,
       12.467, 12.5  , 12.533, 12.567, 12.6  , 12.633, 12.667, 12.7  ,
       12.742, 12.783, 12.825, 12.867, 12.908, 12.95 , 12.992, 13.033,
       13.075, 13.117, 13.158, 13.2  , 14.183, 15.167, 16.15 , 17.133,
       18.117, 19.1  , 20.083, 21.067, 22.05 , 23.033, 24.017, 25.   ,
       25.   , 25.   , 25.   , 25.   , 25.   , 25.   , 25.   , 25.   ,
       25.   , 25.   , 25.   , 25.   , 24.958, 24.917, 24.875, 24.833,
       24.792, 24.75 , 24.708, 24.667, 24.625, 24.583, 24.542, 24.5  ,
       24.658, 24.817, 24.975, 25.133, 25.292, 25.45 , 25.608, 25.767,
       25.925, 26.083, 26.242, 26.4  , 26.325, 26.25 , 26.175, 26.1  ,
       26.025, 25.95 , 25.875, 25.8  , 25.725, 25.65 , 25.575, 25.5  ,
       25.575, 25.65 , 25.725, 25.8  , 25.875, 25.95 , 26.025, 26.1  ,
       26.175, 26.25 , 26.325, 26.4  , 26.325, 26.25 , 26.175, 26.1  ,
       26.025, 25.95 , 25.875, 25.8  , 25.725, 25.65 , 25.575, 25.5  ,
       25.575, 25.65 , 25.725, 25.8  , 25.875, 25.95 , 26.025, 26.1  ,
       26.175, 26.25 , 26.325, 26.4  , 26.325, 26.25 , 26.175, 26.1  ,
       26.025, 25.95 , 25.875, 25.8  , 25.725, 25.65 , 25.575, 25.5  ,
       25.575, 25.65 , 25.725, 25.8  , 25.875, 25.95 , 26.025, 26.1  ,
       26.175, 26.25 , 26.325, 26.4  , 26.358, 26.317, 26.275, 26.233,
       26.192, 26.15 , 26.108, 26.067, 26.025, 25.983, 25.942, 25.9  ,
       25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  ,
       25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  ,
       25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  ,
       25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  ,
       25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  ,
       25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  , 25.9  ,
       25.867, 25.833, 25.8  , 25.767, 25.733, 25.7  , 25.667, 25.633,
       25.6  , 25.567, 25.533, 25.5  , 24.475, 23.45 , 22.425, 21.4  ,
       20.375, 19.35 , 18.325, 17.3  , 16.275, 15.25 , 14.225, 13.2  ,
       13.158, 13.117, 13.075, 13.033, 12.992, 12.95 , 12.908, 12.867,
       12.825, 12.783, 12.742, 12.7  , 12.7  , 12.7  , 12.7  , 12.7  ,
       12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  ,
       12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  ,
       12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  ,
       12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  , 12.7  ])
(Pdb) --KeyboardInterrupt--
(Pdb) df
   Simulation   Category   Object Parameter                                             Series
0   DAY_AHEAD  Generator  1_HYDRO   PMin MW  [12.7, 13.7, 25.9, 26.9, 39.6, 38.7, 38.2, 39....
1   DAY_AHEAD  Generator   2_RTPV   PMin MW  [0.0, 0.0, 0.0, 0.0, 0.0, 0.1, 1.7, 3.5, 5.2, ...
2   DAY_AHEAD  Generator  1_HYDRO   PMax MW  [12.7, 13.7, 25.9, 26.9, 39.6, 38.7, 38.2, 39....
3   DAY_AHEAD  Generator   2_RTPV   PMax MW  [0.0, 0.0, 0.0, 0.0, 0.0, 0.1, 1.7, 3.5, 5.2, ...
4   DAY_AHEAD  Generator    10_PV   PMax MW  [0.0, 0.0, 0.0, 0.0, 0.0, 9.4, 14.1, 17.2, 19....
5   DAY_AHEAD  Generator   4_WIND   PMax MW  [83.4, 112.4, 106.6, 95.4, 70.1, 49.9, 25.4, 6...
6   DAY_AHEAD       Area        1   MW Load  [41.178, 39.466, 38.575, 38.723, 39.406, 41.37...
7   DAY_AHEAD       Area        2   MW Load  [62.522, 59.348, 57.006, 56.288, 55.344, 56.00...
8   REAL_TIME  Generator  1_HYDRO   PMin MW  [12.7, 12.7, 12.7, 12.7, 12.7, 12.7, 12.7, 12....
9   REAL_TIME  Generator   2_RTPV   PMin MW  [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
10  REAL_TIME  Generator  1_HYDRO   PMax MW  [12.7, 12.7, 12.7, 12.7, 12.7, 12.7, 12.7, 12....
11  REAL_TIME  Generator   2_RTPV   PMax MW  [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
12  REAL_TIME  Generator    10_PV   PMax MW  [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
13  REAL_TIME  Generator   4_WIND   PMax MW  [93.612, 93.208, 96.235, 102.744, 109.69, 116....
14  REAL_TIME       Area        1   MW Load  [39.322, 38.111, 38.694, 37.62, 38.557, 36.919...
15  REAL_TIME       Area        2   MW Load  [63.95, 63.61, 63.346, 63.101, 62.856, 62.592,...
(Pdb) --KeyboardInterrupt--
(Pdb)

Without being familiar with the Egret internals, it looks like this was instead expected to be a pandas.Series (at least from what I understand from https://github.com/grid-parity-exchange/Egret/blob/main/egret/parsers/rts_gmlc/parser.py#L772): if this is the case, it's possible that some change introduced in pandas 1.5 somehow causes numpy arrays to be returned/created instead, which are not compatible with the timestamp-based indexing used by Egret.