Bug related to upcoming DST change?

[g1za]

I'm on the latest version of the add-on and this morning I noticed that the latest processed data was from this night at 2AM. In the log I found the error below (the most recent I have a available); before everything was running smoothly. I have the feeling this may be related to the daylight savings time change happening tonight at 2AM; what do you think?

`2024-03-30 09:12:11,044 - web_server - INFO - Passed runtime parameters: {'prediction_horizon': 48, 'pv_power_forecast': [1085, 1229, 1373, 1478, 1584, 1631, 1678, 1661, 1644, 1602, 1559, 1453, 1347, 1199, 1051, 848, 644, 470, 297, 148, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 301, 602, 814, 1025, 1190], 'num_def_loads': 1, 'P_deferrable_nom': [0], 'def_total_hours': [1], 'treat_def_as_semi_cont': [1], 'set_def_constant': [0], 'def_start_timestep': [0], 'def_end_timestep': [0], 'soc_init': 0.517, 'soc_final': 0.3, 'load_cost_forecast': [0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975], 'prod_price_forecast': [0.08416, 0.08416, 0.05786, 0.05786, 0.03156, 0.03156, 0.033664, 0.033664, 0.038924, 0.038924, 0.017884, 0.017884, 0.030508, 0.030508, 0.071536, 0.071536, 0.095732, 0.095732, 0.09994, 0.09994, 0.106252, 0.106252, 0.104148, 0.104148, 0.09994, 0.09994, 0.09994, 0.09994, 0.093628, 0.093628, 0.102044, 0.102044, 0.102044, 0.102044, 0.102044, 0.102044, 0.09994, 0.09994, 0.09994, 0.09994, 0.102044, 0.102044, 0.09994, 0.09994, 0.103096, 0.103096, 0.09994, 0.09994], 'alpha': 0.25, 'beta': 0.75, 'model_type': 'KNN'}
2024-03-30 09:12:11,044 - web_server - INFO -  >> Setting input data dict
2024-03-30 09:12:11,044 - web_server - INFO - Setting up needed data
2024-03-30 09:12:11,045 - web_server - INFO - Retrieve hass get data method initiated...
2024-03-30 09:12:13,279 - web_server - INFO - Retrieving weather forecast data using method = list
2024-03-30 09:12:13,280 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-03-30 09:12:13,280 - web_server - INFO - Retrieve hass get data method initiated...
2024-03-30 09:12:17,792 - web_server - INFO -  >> Performing naive MPC optimization...
2024-03-30 09:12:17,793 - web_server - INFO - Performing naive MPC optimization
2024-03-30 09:12:17,795 - web_server - ERROR - Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 141, in action_call
    opt_res = naive_mpc_optim(input_data_dict, app.logger)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 273, in naive_mpc_optim
    df_input_data_dayahead = input_data_dict['fcst'].get_load_cost_forecast(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 711, in get_load_cost_forecast
    forecast_out = self.get_forecast_out_from_csv(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 526, in get_forecast_out_from_csv
    forecast_out = pd.DataFrame(
                   ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py", line 758, in __init__
    mgr = ndarray_to_mgr(
          ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 337, in ndarray_to_mgr
    _check_values_indices_shape_match(values, index, columns)
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 408, in _check_values_indices_shape_match
    raise ValueError(f"Shape of passed values is {passed}, indices imply {implied}")
ValueError: Shape of passed values is (32, 1), indices imply (30, 1)`

[g1za]

If useful I see some indices change over time

`2024-03-30 10:30:00,856 - web_server - INFO - Passed runtime parameters: {'prediction_horizon': 48, 'pv_power_forecast': [1478, 1584, 1631, 1678, 1661, 1644, 1602, 1559, 1453, 1347, 1199, 1051, 848, 644, 470, 297, 148, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 301, 602, 814, 1025, 1190, 1356, 1546, 1737], 'num_def_loads': 1, 'P_deferrable_nom': [0], 'def_total_hours': [1], 'treat_def_as_semi_cont': [1], 'set_def_constant': [0], 'def_start_timestep': [0], 'def_end_timestep': [0], 'soc_init': 0.562, 'soc_final': 0.3, 'load_cost_forecast': [0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975], 'prod_price_forecast': [0.05786, 0.03156, 0.03156, 0.033664, 0.033664, 0.038924, 0.038924, 0.017884, 0.017884, 0.030508, 0.030508, 0.071536, 0.071536, 0.095732, 0.095732, 0.09994, 0.09994, 0.106252, 0.106252, 0.104148, 0.104148, 0.09994, 0.09994, 0.09994, 0.09994, 0.093628, 0.093628, 0.102044, 0.102044, 0.102044, 0.102044, 0.102044, 0.102044, 0.09994, 0.09994, 0.09994, 0.09994, 0.102044, 0.102044, 0.09994, 0.09994, 0.103096, 0.103096, 0.09994, 0.09994, 0.08416, 0.08416, 0.05786], 'alpha': 0.25, 'beta': 0.75, 'model_type': 'KNN'}
2024-03-30 10:30:00,856 - web_server - INFO -  >> Setting input data dict
2024-03-30 10:30:00,856 - web_server - INFO - Setting up needed data
2024-03-30 10:30:00,858 - web_server - INFO - Retrieve hass get data method initiated...
2024-03-30 10:30:03,126 - web_server - INFO - Retrieving weather forecast data using method = list
2024-03-30 10:30:03,126 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-03-30 10:30:03,127 - web_server - INFO - Retrieve hass get data method initiated...
2024-03-30 10:30:07,595 - web_server - INFO -  >> Performing naive MPC optimization...
2024-03-30 10:30:07,595 - web_server - INFO - Performing naive MPC optimization
2024-03-30 10:30:07,598 - web_server - ERROR - Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 141, in action_call
    opt_res = naive_mpc_optim(input_data_dict, app.logger)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 273, in naive_mpc_optim
    df_input_data_dayahead = input_data_dict['fcst'].get_load_cost_forecast(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 711, in get_load_cost_forecast
    forecast_out = self.get_forecast_out_from_csv(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 526, in get_forecast_out_from_csv
    forecast_out = pd.DataFrame(
                   ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py", line 758, in __init__
    mgr = ndarray_to_mgr(
          ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 337, in ndarray_to_mgr
    _check_values_indices_shape_match(values, index, columns)
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 408, in _check_values_indices_shape_match
    raise ValueError(f"Shape of passed values is {passed}, indices imply {implied}")
ValueError: Shape of passed values is (29, 1), indices imply (27, 1)`

[g1za]

`2024-03-30 13:30:00,857 - web_server - INFO - Passed runtime parameters: {'prediction_horizon': 48, 'pv_power_forecast': [1602, 1551, 1445, 1339, 1190, 1042, 843, 644, 470, 297, 148, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 301, 602, 814, 1025, 1190, 1356, 1546, 1737, 1902, 2067, 2016, 1966, 1834, 1703], 'num_def_loads': 1, 'P_deferrable_nom': [0], 'def_total_hours': [1], 'treat_def_as_semi_cont': [1], 'set_def_constant': [0], 'def_start_timestep': [0], 'def_end_timestep': [0], 'soc_init': 0.71, 'soc_final': 0.3, 'load_cost_forecast': [0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975], 'prod_price_forecast': [0.038924, 0.017884, 0.017884, 0.030508, 0.030508, 0.071536, 0.071536, 0.095732, 0.095732, 0.09994, 0.09994, 0.106252, 0.106252, 0.104148, 0.104148, 0.09994, 0.09994, 0.09994, 0.09994, 0.093628, 0.093628, 0.097836, 0.097836, 0.09994, 0.09994, 0.09994, 0.09994, 0.09994, 0.09994, 0.09994, 0.09994, 0.09994, 0.09994, 0.100992, 0.100992, 0.100992, 0.100992, 0.098888, 0.098888, 0.095732, 0.095732, 0.091524, 0.091524, 0.06312, 0.06312, 0.04208, 0.04208, 0.003156], 'alpha': 0.25, 'beta': 0.75, 'model_type': 'KNN'}
2024-03-30 13:30:00,858 - web_server - INFO -  >> Setting input data dict
2024-03-30 13:30:00,858 - web_server - INFO - Setting up needed data
2024-03-30 13:30:00,859 - web_server - INFO - Retrieve hass get data method initiated...
2024-03-30 13:30:03,069 - web_server - INFO - Retrieving weather forecast data using method = list
2024-03-30 13:30:03,069 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-03-30 13:30:03,070 - web_server - INFO - Retrieve hass get data method initiated...
2024-03-30 13:30:07,244 - web_server - INFO -  >> Performing naive MPC optimization...
2024-03-30 13:30:07,244 - web_server - INFO - Performing naive MPC optimization
2024-03-30 13:30:07,246 - web_server - ERROR - Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 141, in action_call
    opt_res = naive_mpc_optim(input_data_dict, app.logger)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 273, in naive_mpc_optim
    df_input_data_dayahead = input_data_dict['fcst'].get_load_cost_forecast(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 711, in get_load_cost_forecast
    forecast_out = self.get_forecast_out_from_csv(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 526, in get_forecast_out_from_csv
    forecast_out = pd.DataFrame(
                   ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py", line 758, in __init__
    mgr = ndarray_to_mgr(
          ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 337, in ndarray_to_mgr
    _check_values_indices_shape_match(values, index, columns)
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 408, in _check_values_indices_shape_match
    raise ValueError(f"Shape of passed values is {passed}, indices imply {implied}")
ValueError: Shape of passed values is (23, 1), indices imply (21, 1)`

[g1za]

`2024-03-30 14:00:02,529 - web_server - INFO - Passed runtime parameters: {'prediction_horizon': 48, 'pv_power_forecast': [1551, 1445, 1339, 1190, 1042, 843, 644, 470, 297, 148, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 301, 602, 814, 1025, 1190, 1356, 1546, 1737, 1902, 2067, 2016, 1966, 1834, 1703, 1546], 'num_def_loads': 1, 'P_deferrable_nom': [0], 'def_total_hours': [1], 'treat_def_as_semi_cont': [1], 'set_def_constant': [0], 'def_start_timestep': [0], 'def_end_timestep': [0], 'soc_init': 0.721, 'soc_final': 0.3, 'load_cost_forecast': [0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.114969, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975, 0.099975], 'prod_price_forecast': [0.017884, 0.017884, 0.030508, 0.030508, 0.071536, 0.071536, 0.095732, 0.095732, 0.09994, 0.09994, 0.106252, 0.106252, 0.104148, 0.104148, 0.09994, 0.09994, 0.09994, 0.09994, 0.093628, 0.093628, 0.097836, 0.097836, 0.09994, 0.09994, 0.09994, 0.09994, 0.09994, 0.09994, 0.09994, 0.09994, 0.09994, 0.09994, 0.100992, 0.100992, 0.100992, 0.100992, 0.098888, 0.098888, 0.095732, 0.095732, 0.091524, 0.091524, 0.06312, 0.06312, 0.04208, 0.04208, 0.003156, 0.003156], 'alpha': 0.25, 'beta': 0.75, 'model_type': 'KNN'}
2024-03-30 14:00:02,530 - web_server - INFO -  >> Setting input data dict
2024-03-30 14:00:02,530 - web_server - INFO - Setting up needed data
2024-03-30 14:00:02,531 - web_server - INFO - Retrieve hass get data method initiated...
2024-03-30 14:00:04,724 - web_server - INFO - Retrieving weather forecast data using method = list
2024-03-30 14:00:04,725 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-03-30 14:00:04,725 - web_server - INFO - Retrieve hass get data method initiated...
2024-03-30 14:00:09,193 - web_server - INFO -  >> Performing naive MPC optimization...
2024-03-30 14:00:09,193 - web_server - INFO - Performing naive MPC optimization
2024-03-30 14:00:09,195 - web_server - ERROR - Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 141, in action_call
    opt_res = naive_mpc_optim(input_data_dict, app.logger)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 273, in naive_mpc_optim
    df_input_data_dayahead = input_data_dict['fcst'].get_load_cost_forecast(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 711, in get_load_cost_forecast
    forecast_out = self.get_forecast_out_from_csv(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 526, in get_forecast_out_from_csv
    forecast_out = pd.DataFrame(
                   ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py", line 758, in __init__
    mgr = ndarray_to_mgr(
          ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 337, in ndarray_to_mgr
    _check_values_indices_shape_match(values, index, columns)
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 408, in _check_values_indices_shape_match
    raise ValueError(f"Shape of passed values is {passed}, indices imply {implied}")
ValueError: Shape of passed values is (22, 1), indices imply (20, 1)`

[poupaerta]

I am from Belgium and this morning I also got an error message. Yesterday it workt perfectly.

2024-03-30 14:25:18,524 - web_server - INFO - Passed runtime parameters: {} 2024-03-30 14:25:18,524 - web_server - INFO - >> Setting input data dict 2024-03-30 14:25:18,525 - web_server - INFO - Setting up needed data 2024-03-30 14:25:18,530 - web_server - ERROR - Exception on /action/dayahead-optim [POST] Traceback (most recent call last): File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app response = self.full_dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request rv = self.handle_user_exception(e) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request rv = self.dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(view_args) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 104, in action_call input_data_dict = set_input_data_dict(config_path, str(data_path), costfun, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 64, in set_input_data_dict fcst = Forecast(retrieve_hass_conf, optim_conf, plant_conf, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 162, in init freq=self.freq).round(self.freq, ambiguous='infer', nonexistent=self.freq) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/pandas/core/indexes/extension.py", line 98, in method result = attr(self._data, *args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/pandas/core/arrays/datetimelike.py", line 2026, in round return self._round(freq, RoundTo.NEAREST_HALF_EVEN, ambiguous, nonexistent) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/pandas/core/arrays/datetimelike.py", line 2002, in _round return result.tz_localize( ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/pandas/core/arrays/_mixins.py", line 86, in method return meth(self, args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/pandas/core/arrays/datetimes.py", line 1040, in tz_localize new_dates = tzconversion.tz_localize_to_utc( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "pandas/_libs/tslibs/tzconversion.pyx", line 380, in pandas._libs.tslibs.tzconversion.tz_localize_to_utc ValueError: The provided timedelta will relocalize on a nonexistent time: 0 days 00:30:00

[MikaelHoogen]

Try 'prediction_horizon': 46

[adammcdonagh]

I'm also getting failures. Just updated to the latest version today. Different error though. DST starts tonight.

2024-03-30 16:07:33,940 - web_server - INFO - Passed runtime parameters: {'prediction_horizon': 48, 'pv_power_forecast': [936, 519, 304, 162, 71, 18, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 21, 72, 153, 294, 486, 657, 762, 762, 714, 660, 639, 651, 645, 621, 564, 489, 411, 336, 255], 'soc_init': 0.93}
2024-03-30 16:07:33,940 - web_server - INFO -  >> Setting input data dict
2024-03-30 16:07:33,941 - web_server - INFO - Setting up needed data
2024-03-30 16:07:33,944 - web_server - INFO - Retrieve hass get data method initiated...
2024-03-30 16:07:34,244 - web_server - INFO - Retrieving weather forecast data using method = list
2024-03-30 16:07:34,246 - web_server - INFO - Retrieving data from hass for load forecast using method = mlforecaster
2024-03-30 16:07:34,246 - web_server - INFO - Retrieve hass get data method initiated...
2024-03-30 16:07:34,834 - web_server - ERROR - Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 104, in action_call
    input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 127, in set_input_data_dict
    P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'], set_mix_forecast=True, df_now=df_input_data)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 626, in get_load_forecast
    data = pd.DataFrame.from_dict(data_dict)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py", line 1760, in from_dict
    return cls(data, index=index, columns=columns, dtype=dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py", line 709, in __init__
    mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 481, in dict_to_mgr
    return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 115, in arrays_to_mgr
    index = _extract_index(arrays)
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 655, in _extract_index
    raise ValueError("All arrays must be of the same length")
ValueError: All arrays must be of the same length

[davidusb-geek]

The original error message from @g1za seems related to the passed data. The error is a checkup done on input data. So not much to do here. You are not passing the correct length of those data lists

[g1za]

I thought that could be a reason (maybe my source data had changes due to the time change), so I checked but according to the command I'm expected to pass 48 elements ('prediction_horizon': 48) and pv_power_forecast, 'load_cost_forecast', 'prod_price_forecast' are all arrays of 48 elements (see latest command for example). I'm using naive mode and I have way more than one day of historical data for my house load.

May the problem be related to data pull from HA past data?

Edit: I also do not understand why the error has a decreasing value for expected data; each time step this is decreased by 1 and it seems I'm always passing 2 more elements than I should (while for me it is always 48): `

[...]
2024-03-30 13:30: Shape of passed values is (23, 1), indices imply (21, 1)
2024-03-30 14:00: Shape of passed values is (22, 1), indices imply (20, 1)
[...]
2024-03-30 23:00: Shape of passed values is (4, 1), indices imply (2, 1)
2024-03-30 23:30: Shape of passed values is (2, 1), indices imply (1, 1) (<-- yes, this is what is reported in the log, this time shape of passed values is -2)
2024-03-31 00:00: Shape of passed values is (48, 1), indices imply (46, 1) (<-- it seems the two 30' steps of the missing hour generated by DST are missing here)
2024-03-31 00:30: Shape of passed values is (47, 1), indices imply (45, 1)
2024-03-31 01:00: Shape of passed values is (46, 1), indices imply (44, 1)
2024-03-31 01:30: Shape of passed values is (45, 1), indices imply (43, 1)

`

[g1za]

I definitely believe it is still a bug related to DST, @davidusb-geek At the switch to the new time the log didn't show any error anymore and the optimization was performed correctly. Please see below and my previous comment.

`2024-03-31 03:00:01,574 - web_server - INFO - Passed runtime parameters: {'prediction_horizon': 48, 'pv_power_forecast': [0, 0, 0, 0, 0, 0, 0, 301, 602, 810, 1017, 1174, 1330, 1432, 1534, 1593, 1652, 1656, 1661, 1618, 1576, 1474, 1373, 1229, 1085, 886, 686, 470, 254, 170, 85, 42, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'num_def_loads': 1, 'P_deferrable_nom': [0], 'def_total_hours': [1], 'treat_def_as_semi_cont': [1], 'set_def_constant': [0], 'def_start_timestep': [0], 'def_end_timestep': [0], 'soc_init': 0.368, 'soc_final': 0.3, 'load_cost_forecast': [0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341, 0.100341], 'prod_price_forecast': [0.09994, 0.09994, 0.09994, 0.09994, 0.09994, 0.09994, 0.100992, 0.100992, 0.100992, 0.100992, 0.098888, 0.098888, 0.095732, 0.095732, 0.091524, 0.091524, 0.06312, 0.06312, 0.04208, 0.04208, 0.003156, 0.003156, 0.013676, 0.013676, 0.062068, 0.062068, 0.09994, 0.09994, 0.107304, 0.107304, 0.130448, 0.130448, 0.140968, 0.140968, 0.133604, 0.133604, 0.116772, 0.116772, 0.104148, 0.104148, 0.093628, 0.093628, 0.097836, 0.097836, 0.09994, 0.09994, 0.09994, 0.09994], 'alpha': 0.25, 'beta': 0.75, 'model_type': 'KNN'}
2024-03-31 03:00:01,574 - web_server - INFO -  >> Setting input data dict
2024-03-31 03:00:01,574 - web_server - INFO - Setting up needed data
2024-03-31 03:00:01,576 - web_server - INFO - Retrieve hass get data method initiated...
2024-03-31 03:00:03,759 - web_server - INFO - Retrieving weather forecast data using method = list
2024-03-31 03:00:03,760 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-03-31 03:00:03,760 - web_server - INFO - Retrieve hass get data method initiated...
2024-03-31 03:00:08,038 - web_server - INFO -  >> Performing naive MPC optimization...
2024-03-31 03:00:08,039 - web_server - INFO - Performing naive MPC optimization
2024-03-31 03:00:08,045 - web_server - INFO - Perform an iteration of a naive MPC controller
2024-03-31 03:00:08,052 - web_server - DEBUG - Deferrable load 0: Proposed optimization window: 0 --> 0
2024-03-31 03:00:08,052 - web_server - DEBUG - Deferrable load 0: Validated optimization window: 0 --> 0
2024-03-31 03:00:08,135 - web_server - INFO - Status: Optimal
2024-03-31 03:00:08,135 - web_server - INFO - Total value of the Cost function = -686.33`

[davidusb-geek]

Thanks @g1za for looking at this. It is definitely not 100% DST change proof. The problem here is that I've not been able to reproduce this exact situation in testing. But we will need to try another round. Great to hear that it's working fine now

[vermut]

That is 100% DST related, because out of all 365 days I picked yesterday to finally configure this beast and spent 6 hours digging through various numpy shapes and csv_load exceptions. Of course it immediately worked today from second try.

[adammcdonagh]

Mine is still failing. My guess is that until it’s 48hrs after the DST change it’s going to keep getting 1 hour less from history, and complain that the values don’t match up. Max I can do at present is a horizon of 12 for some reason.

[davidusb-geek]

Yes we still need to work this one out

[werdnum]

I'm getting a similar problem with the upcoming change in Sydney, but in this case an AmbiguousTimeError is being thrown.

https://hastebin.com/share/awanuqoduk.python

[werdnum]

If I can make a suggestion, in my experience the best strategy for anything time related is to work in UTC as much as possible and convert to and from the local time zone at the API boundary (if necessary) or for display to the user.