davidusb-geek / emhass

emhass: Energy Management for Home Assistant, is a Python module designed to optimize your home energy interfacing with Home Assistant.
MIT License
260 stars 51 forks source link

EMHASS cannot retrieve load history from HA sensor #257

Closed Octofinger closed 2 months ago

Octofinger commented 2 months ago

Describe the bug EMHASS naive-mpc-optim fails because it can't read my load sensor. This just started one early morning without me having made any changes to neither configuration, nor made any restarts.

The logs show the following error message:

2024-04-14 19:35:00,617 - web_server - INFO - Passed runtime parameters: {'prediction_horizon': 24, 'soc_init': 0.05, 'soc_final': 0.5, 'def_total_hours': [], 'load_cost_forecast': [0.55, 0.37, 0.35, 0.35, 0.35, 0.32, 0.33, 0.34, 0.35, 0.35, 0.59, 0.94, 2.45, 1.86, 1.07, 1.07, 1.01, 0.97, 0.83, 0.63, 0.78, 0.87, 1.0, 0.99], 'prod_price_forecast': [0.32, 0.14, 0.12, 0.12, 0.12, 0.09, 0.1, 0.11, 0.12, 0.12, 0.36, 0.71, 2.22, 1.63, 0.84, 0.84, 0.78, 0.74, 0.6, 0.4, 0.55, 0.64, 0.77, 0.76], 'pv_power_forecast': [109, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 222, 567, 1174, 2092, 3042, 4154, 5169, 6031, 6454, 6285, 5663, 4634, 2908]}
2024-04-14 19:35:00,617 - web_server - INFO -  >> Setting input data dict
2024-04-14 19:35:00,617 - web_server - INFO - Setting up needed data
2024-04-14 19:35:00,620 - web_server - INFO - Retrieve hass get data method initiated...
2024-04-14 19:35:02,155 - web_server - INFO - Retrieving weather forecast data using method = list
2024-04-14 19:35:02,157 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-04-14 19:35:02,157 - web_server - INFO - Retrieve hass get data method initiated...
2024-04-14 19:35:02,227 - web_server - ERROR - The retrieved JSON is empty, A sensor:sensor.template_emhass_no_var_load_2 may have 0 days of history or passed sensor may not be correct
2024-04-14 19:35:02,229 - web_server - ERROR - Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 108, in action_call
    input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 128, in set_input_data_dict
    df_input_data_dayahead = pd.concat([P_PV_forecast, P_load_forecast], axis=1)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/reshape/concat.py", line 372, in concat
    op = _Concatenator(
         ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/reshape/concat.py", line 462, in __init__
    raise TypeError(msg)
TypeError: cannot concatenate object of type '<class 'bool'>'; only Series and DataFrame objs are valid

To Reproduce This happens every time I try to call the optimization.

Expected behavior EMHASS should be able to read the sensor's value. It clearly has a history if I look at it in HA: image

Screenshots If applicable, add screenshots to help explain your problem.

Home Assistant installation type

Your hardware

EMHASS installation type

Additional context I have tested retrieving the sensor history using curl from within the emhass container using the same HA URL and access token as specified in secrets_emhass.yaml and it works well. I have tried creating a second load sensor in HA, but that cannot be read either by EMHASS. I have tried using the sensor in an ML forecaster fit call from EMHASS web UI, but only with the same result: image

Octofinger commented 2 months ago

I ran this Python script to test the retrieval of history. I ran it from a separate PC on the same network. It returns a complete list of sensor history, so still can't reproduce the EMHASS behavior. (secrets masked)

import pandas as pd
from requests import get, post

days_list = pd.date_range(end='2024-04-17', freq='D', periods=5)
print(days_list)
print(days_list[0].isoformat())
day = days_list [0]
var = 'sensor.template_emhass_no_var_load_2'
hass_url= 'https://my.homeassistant.url:8123/'
long_lived_token = 'XXXXXXXXXXXXXXXX'

url = hass_url+"api/history/period/"+day.isoformat()+"?filter_entity_id="+var

headers = {

    "Authorization": "Bearer " + long_lived_token,
    "content-type": "application/json",
}

# Then once that all this is defined there is no much to it than this:
response = get(url, headers=headers)

# And extract the data >> This is where your code is failing so check this part:
data = response.json()[0]

# If successful then print
print(data)
[{'entity_id': 'sensor.template_emhass_no_var_load_2', 'state': '694.05', 'attributes': {'unit_of_measurement': 'W', 'device_class': 'power'}, 'last_changed': '2024-04-12T22:00:00.928736+00:00', 'last_updated': '2024-04-12T22:00:00.928736+00:00', 'context': {'id': '01HVG824J3QD58P2P1KHNK5EBW', 'parent_id': None, 'user_id': None}}, {'entity_id': 'sensor.template_emhass_no_var_load_2', 'state': '693.03', 'attributes': {'unit_of_measurement': 'W', 'device_class': 'power'}, 'last_changed': '2024-04-12T22:00:02.861105+00:00', 'last_updated': '2024-04-12T22:00:02.861105+00:00', 'context': {'id': '01HVG824J3QD58P2P1KHNK5EBW', 'parent_id': None, 'user_id': None}}, {'entity_id': 'sensor.template_emhass_no_var_load_2', 'state': '694.55', 'attributes': {'unit_of_measurement': 'W', 'device_class': 'power'}, 'last_changed': '2024-04-12T22:00:05.114414+00:00', 'last_updated': '2024-04-12T22:00:05.114414+00:00', 'context': {'id': '01HVG824J3QD58P2P1KHNK5EBW', 'parent_id': None, 'user_id': None}}, {'entity_id': 'sensor.template_emhass_no_var_load_2', 'state': '685.56', 'attributes': {'unit_of_measurement': 'W', 'device_class': 'power'}, 'last_changed': '2024-04-12T22:00:17.870891+00:00', 'last_updated': '2024-04-12T22:00:17.870891+00:00', 'context': {'id': '01HVG824J3QD58P2P1KHNK5EBW', 'parent_id': None, 'user_id': None}}, {'entity_id': 'sensor.template_emhass_no_var_load_2', 'state': '684.04', 'attributes': {'unit_of_measurement': 'W', 'device_class': 'power'}, 'last_changed': '2024-04-12T22:00:20.125773+00:00', 'last_updated': '2024-04-12T22:00:20.125773+00:00', 'context': {'id': '01HVG824J3QD58P2P1KHNK5EBW', 'parent_id': None, 'user_id': None}}, {'entity_id': 'sensor.template_emhass_no_var_load_2', 'state': '689.17', 'attributes': {'unit_of_measurement': 'W', 'device_class': 'power'}, 'last_changed': '2024-04-12T22:00:32.871763+00:00', 'last_updated': '2024-04-12T22:00:32.871763+00:00', 'context': {'id': '01HVG824J3QD58P2P1KHNK5EBW', 'parent_id': None, 'user_id': None}}, 
...
davidusb-geek commented 2 months ago

I need to find the time to try to replicate this. In the meantime can you provide the logs from the add-on after a fresh restart of the add-on? It's just to check the information printed on the logger about how data is retrieved from HA. It is true that HA core and EMHASS on docker standalone mode are the least common installation methods these days.

Octofinger commented 2 months ago

OMG, now the problem is gone. I haven't done anything. No restarts, no config changes, no nothing... I seriously have no friggin idea what happened. The log errors are gone and the optimization and data publisher runs without issues.

I'll close this. If anything like this happens again, I'll try to set up a dev environment with a clone of this repo to try to get more details before posting anything.

Thanks a million for your engagement in this. emhass is key to making my solar facility more profitable and I really appreciate this. I bought you some coffee.