Feature request: Make own forecast data persistent

[overas]

After the load_cost_forecast and prod_price_forecast from Nordpool addon are passed as a list to emhass the data is no persistent. When restarting emhass the data is lost and the default values are used in unit_load_cast and unit_prod_price.

This cause problems: When I use the service: Shell Command: post_nordpool_forecast to update the data, the data do not exist because Nordpool are only updated 13:00 and are only updated once from Nordpool. or The data exist, but today and tomorrow data from post_nordpool_forecast are not correct placed in time Example: Tomorrow price data - 0.351 are placed in 21:00-22:00, but the correct time is 00:00-01:00.

It would have been nice if the forecast data could have been stored in emhass and not lost when emhass are restarted.

[davidusb-geek]

This seems complicated, management of data and awareness of the code to add-on restarts can be hard. The code is already designed to survive restarts and retain your latest optimization results so that you can control your deferrable loads properly. I think that on most cases this is enough. I don't think that I'm willing to spend a lot of time to find an efficient solution to this. However pull requests are more than welcomed. This also seems like something that can be addressed from outside emhass. Just create a sensor that retains the correct forecast and pass that manually in the event of a restart. You will still have this problem and even more complicated to handle inside emhass in the case of an add-on update. Much more easy to handle it from the outside of emhass. What I do that is also a possible solution for you is that when I know that I will restart HA or in the case of an add-on update, then I switch back the control of my deferrable loads to an automated mode that just control the loads based on a static schedule. The next day I can then switch back to use the emhass optimization results again.

[overas]

Thank you for the good feedback David. I am going to investigate the possible solution to create a sensor that retain the correct forecast and pass that after a restart. But then I have the problem that the data exist, but today and tomorrow data from post_nordpool_forecast are not correct placed in time Example: Tomorrow price data - 0.351 are placed in 21:00-22:00, but the correct time is 00:00-01:00. How can I solve this problem?

What I do that is also a possible solution for you is that when I know that I will restart HA or in the case of an add-on update, then I switch back the control of my deferrable loads to an automated mode that just control the loads based on a static schedule. The next day I can then switch back to use the emhass optimization results again.

This possible solution could also work. But the easiest solution is to solve it as a automation in Home Assistant. Thanks for your hard word with emhass.

[davidusb-geek]

The offset on the data time slots is recurrent or is just after a restart/update of the add-on? EMHASS is using the timezone of your HA instance. If you're passing your data as a list then the first passed value is considered as the next time slot value. There is no additional treatment of the timestamps. So the passed is defined as it is on the forecast sensor attribute.

[overas]

The offset on the data time slots is recurrent or is just after a restart/update of the add-on?

It is just after a restart/update of the add-on. I triggered the service now at 12:55 with the service. The price data for tomorrow are inserted at nearest time 13:00.

  post_nordpool_forecast:
    'curl -i -H ''Content-Type: application/json'' -X POST -d ''{"load_cost_forecast":{{(
    (state_attr(''sensor.nordpool'', ''raw_tomorrow'')|map(attribute=''value'')|list)[:24])
    }},"prod_price_forecast":{{(
    (state_attr(''sensor.nordpool'', ''raw_tomorrow'')|map(attribute=''value'')|list)[:24])}}}'' http://localhost:5000/action/dayahead-optim'

[2022-09-18 12:55:57,654] INFO in command_line: Setting up needed data
[2022-09-18 12:55:57,660] INFO in forecast: Retrieving weather forecast data using method = scrapper
[2022-09-18 12:56:01,037] INFO in forecast: Retrieving data from hass for load forecast using method = naive
[2022-09-18 12:56:01,039] INFO in retrieve_hass: Retrieve hass get data method initiated...
[2022-09-18 12:56:05,196] INFO in web_server:  >> Performing dayahead optimization...
[2022-09-18 12:56:05,197] INFO in command_line: Performing day-ahead forecast optimization
[2022-09-18 12:56:05,241] INFO in optimization: Perform optimization for the day-ahead
[2022-09-18 12:56:05,356] INFO in optimization: Status: Optimal
[2022-09-18 12:56:05,357] INFO in optimization: Total value of the Cost function = -88.39

[overas]

create a sensor that retains the correct forecast and pass that manually in the event of a restart

Nordpool data always starts at midnight, the first element in the list will always refer to 01:00, the second element 02:00. When inserting the price data I must ensure the data fed to EMHASS referes to the correct time slot. So if I trigger to restore the sensor state/attribute after a Emhass restart at 09:45, it is expecting to see the prices for 10:00, 11:00, and so on. How can I do this with template?

[davidusb-geek]

Yes I don't pass this kind of sensor data so I can't provide you with a ready to use template, however this seems doable with a template. You need to find a way to shift your list of values to the correct amount of time slots according to the current time of the day. If you find a solution for this I will be interested to put this on the documentation to help others. Correct me if I'm wrong but you seem to be passing the exact same data for load_cost_forecast and prod_price_forecast

[overas]

After reading the nordpool post in HA forum I found a template I can use. I going to share the solution if I can automate a update when emhass restart.

{% set today_price = state_attr('sensor.nordpool', 'today') %}
{{ today_price[now().hour:] }}

template:
  - sensor:
      - name: "Emhass forecast price update"
        unit_of_measurement: NOK/kWh
        icon: mdi:flash
        state: >
          {% set today_price = state_attr('sensor.nordpool', 'today') %}
          {{ today_price[now().hour:] }}

Correct me if I'm wrong but you seem to be passing the exact same data for load_cost_forecast and prod_price_forecast

This is correct. My powercompany Tibber pay the same for my electricity in the event of overproduction as I would have to pay on the market. They buy my profit, and I get paid Nord Pool's spot price hour by hour.

[overas]

The shell_command passes the list with forecast values to emhass, but has only values from 19:00 to 00:00. Emhass says the passed data should be 24, but it is only five in the list. Is this something you can solve?

shell_command:
  forecast_price_update:
    'curl -i -H ''Content-Type: application/json'' -X POST -d ''{"load_cost_forecast":{{(
    states(''sensor.emhass_forecast_price_update''))}}}'' http://localhost:5000/action/dayahead-optim'

[2022-09-18 19:41:38,473] INFO in web_server:  >> Performing dayahead optimization...
[2022-09-18 19:41:38,473] INFO in command_line: Performing day-ahead forecast optimization
[2022-09-18 19:41:38,481] INFO in optimization: Perform optimization for the day-ahead
[2022-09-18 19:41:38,592] INFO in optimization: Status: Optimal
[2022-09-18 19:41:38,593] INFO in optimization: Total value of the Cost function = -4.06
[2022-09-18 19:41:56,783] INFO in command_line: Setting up needed data
[2022-09-18 19:41:56,786] ERROR in utils: ERROR: The passed data is either not a list or the length is not correct, length should be 24
[2022-09-18 19:41:56,787] ERROR in utils: Passed type is <class 'list'> and length is 24

[davidusb-geek]

Yes but those are just "today" values right? Well you just need to concatenate those with the "tomorrow" values. Maybe something like this?

state: >
          {% set today_price = state_attr('sensor.nordpool', 'today') %}
          {% set tomorrow_price = state_attr('sensor.nordpool', 'tomorrow') %}
          {{ today_price[now().hour:]+tomorrowwprice }}

[lovekull76]

This seems complicated, management of data and awareness of the code to add-on restarts can be hard. The code is already designed to survive restarts and retain your latest optimization results so that you can control your deferrable loads properly. I think that on most cases this is enough. I don't think that I'm willing to spend a lot of time to find an efficient solution to this. However pull requests are more than welcomed. This also seems like something that can be addressed from outside emhass. Just create a sensor that retains the correct forecast and pass that manually in the event of a restart. You will still have this problem and even more complicated to handle inside emhass in the case of an add-on update. Much more easy to handle it from the outside of emhass. What I do that is also a possible solution for you is that when I know that I will restart HA or in the case of an add-on update, then I switch back the control of my deferrable loads to an automated mode that just control the loads based on a static schedule. The next day I can then switch back to use the emhass optimization results again.

Hi,

Love the add-on but it will not "survive restarts and retain your latest optimization results" in my case, what could be wrong? At reboot/reload of the add-in the default optimization data (your example data) is loaded :(

Best regards, L

[davidusb-geek]

Yes you are right. The add-on is not retraining the saved data because that's just how containers works. I did designed the code to survive restarts by saving the csv data and it works fine with the legacy installation method as a simple Python module. But when I wrote that answer I wasn't aware of this problem because of my lack of experience with docker containers. So this is still a problem and I was wrong with that answer. The current solution is to relaunch the optimization when detecting a restart. It works fine for me and its perfect if you use MPC.

[lovekull76]

Thanks! That explains it. The problem is that I can't just reload and get back the same results. EMHASS is doing a new optimization with no respect to current state of the battery and before 13:00 I don't even have hourly price data for 24h ahead.

//L

Love Kull +46(0)70/7925269

Den mån 23 jan. 2023 kl 00:28 skrev David @.***>:

Yes you are right. The add-on is not retraining the saved data because that's just how containers works. I did designed the add-on to survive restarts by saving the csv data and it works fine with the legacy installation method as a simple Python module. But when I wrote that answer I wasn't aware of this problem because of my lack of experience with docker containers. So this is still a problem and I was wrong with that answer. The current solution is to relaunch the optimization when detecting a restart. It works fine for me and its perfect if you use MPC.

— Reply to this email directly, view it on GitHub https://github.com/davidusb-geek/emhass/issues/20#issuecomment-1399640217, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5ADIOJHAYOKLVG7WYXVVRDWTW7AZANCNFSM6AAAAAAQPEB6YE . You are receiving this because you commented.Message ID: @.***>

[davidusb-geek]

I'm reopening this as we need to figure out a good solution for this.

[lovekull76]

Great! Glad to help if I can!

[lovekull76]

PS. I found the forcast data for battery power in the attributes for sensor.p_batt_forcast, if that data is persistent during reloads one could use that for future updates of the state of said entity maybe. It would actually be feasible to subtract the data from the attribute of p_bat_forecast right after an optimization and initial publish and use that saved data for controlling the battery, without using publish in EMHASS. I might go that route for now until this is solved the official way :)

[davidusb-geek]

That was one of the possible solutions. If you save that sensor on the Home Assistant recorder then just directly use that data which is available after a restart to autoamte your loads (or your battery in your case). That is a simple and nice solution. Still we should be able to find a solution to retrieve a CSV file stored on a persistent folder, I will test that what should be the official solution to this.

[lovekull76]

That would be the right way to do it I guess.

the sensor.p_batt_forecast is actually gone after a HA reload :( But the optimization results are still there and valid as long as I don't re-run the optimization. If doing just a publish I get the bat sensor back with the last optimization data (need to do that on the hour thou otherwise the timing gets off and state of the battery is set to next hour power data).

I guess the add-on is not reloaded when I reload HA?

Can I retrieve the bat data from the Recorder when the sensor is missing?

[davidusb-geek]

I've just released a new version that should survive restarts. Please test and give a feedback if it works. Tested on the development environment and work well. It is just making use of the /data folder that is persistent for add-ons.

[lovekull76]

Great work!

Just installed it and ran a dayahead opt to make sure to populate the new data file at the new location. Webpage showed the new results ok. I reloaded the add-on and the webpage showed no data. I did not do a publish but the entities was still there, I don't know if it used the new opt-data or from the last optim yesterday thou. Did a publish but the entities did not update and no action in the add-on log. I'll investigate this further...

[lovekull76]

PS.

Tested again:

• ran dayahead optim ok
• webpage ok
• publish ok
• reload addon ok
• webpage empty, error
• ran publish, error

s6-rc: info: service s6rc-oneshot-runner: starting s6-rc: info: service s6rc-oneshot-runner successfully started s6-rc: info: service fix-attrs: starting s6-rc: info: service fix-attrs successfully started s6-rc: info: service legacy-cont-init: starting s6-rc: info: service legacy-cont-init successfully started s6-rc: info: service legacy-services: starting services-up: info: copying legacy longrun emhass (no readiness notification) s6-rc: info: service legacy-services successfully started [2023-01-29 09:45:15,047] INFO in web_server: Launching the emhass webserver at: http://0.0.0.0:5000 [2023-01-29 09:45:15,048] INFO in web_server: Home Assistant data fetch will be performed using url: http://supervisor/core/api [2023-01-29 09:45:15,048] INFO in web_server: The data path is: /data [2023-01-29 09:45:15,050] INFO in web_server: Using core emhass version: 0.3.29 [2023-01-29 09:45:39,752] INFO in web_server: EMHASS server online, serving index.html... [2023-01-29 09:45:39,761] WARNING in web_server: Oops.. The data dictionary is empty... Please launch an optimization task [2023-01-29 09:45:43,334] INFO in web_server: EMHASS server online, serving index.html... [2023-01-29 09:45:43,339] WARNING in web_server: Oops.. The data dictionary is empty... Please launch an optimization task [2023-01-29 09:46:33,629] INFO in command_line: Setting up needed data [2023-01-29 09:46:33,670] INFO in web_server: >> Publishing data... [2023-01-29 09:46:33,670] INFO in command_line: Publishing data to HASS instance [2023-01-29 09:46:33,671] ERROR in command_line: File not found error, run an optimization task first. [2023-01-29 09:46:33,671] ERROR in app: Exception on /action/publish-data [POST] Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2525, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1822, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1820, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1796, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 139, in actioncall = publish_data(input_data_dict, app.logger) File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 288, in publish_data idx_closest = opt_res_latest.index.get_indexer([now_precise], method='nearest')[0] UnboundLocalError: local variable 'opt_res_latest' referenced before assignment

[davidusb-geek]

Oh yes I tested it restarting Home Assistant but not just restarting the add-on.

[lovekull76]

OK, did you restart HA only or the hardware (i.e. everything)?

[lovekull76]

Regardless, the csv in /data does not look like it is persistent :/

[lovekull76]

Restarting HA only: webpage OK publish OK entities gone until publish, kind of OK, not an issue if only triggering actions on change in .p_batt.. entity I guess.

Did get this in the log thou:

[2023-01-29 10:45:00,735] INFO in command_line: Setting up needed data [2023-01-29 10:45:00,738] INFO in web_server: >> Publishing data... [2023-01-29 10:45:00,739] INFO in command_line: Publishing data to HASS instance [2023-01-29 10:45:00,775] INFO in retrieve_hass: The received response code is not recognized [2023-01-29 10:45:00,800] INFO in retrieve_hass: The received response code is not recognized [2023-01-29 10:45:00,825] INFO in retrieve_hass: The received response code is not recognized [2023-01-29 10:45:00,852] INFO in retrieve_hass: The received response code is not recognized [2023-01-29 10:45:00,893] INFO in retrieve_hass: The received response code is not recognized [2023-01-29 10:45:00,919] INFO in retrieve_hass: The received response code is not recognized [2023-01-29 10:45:00,941] INFO in retrieve_hass: The received response code is not recognized

[davidusb-geek]

I'm guessing that \data is persistent only for HA restarts and not add-on restarts. So we may need to find a solution for that case. I think that I will test saving to other folders like the \share folder.

[davidusb-geek]

I've just found some bugs implementing the data folder handling. And also I'm now using directly the \share folder. I tested both HA and add-on restarts and it works fine this time! I've just released a new version

[overas]

I get these errors when trying to install Emhass 0.2.27.

Logger: homeassistant.components.websocket_api.http.connection
Source: components/hassio/update.py:169
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 17:44:56 (1 occurrences)
Last logged: 17:44:56

[547242344304] Error updating EMHASS: Can't install davidusb/image-aarch64-emhass:0.2.27: 404 Client Error for http+docker://localhost/v1.41/images/create?tag=0.2.27&fromImage=davidusb%2Fimage-aarch64-emhass&platform=linux%2Farm64: Not Found ("manifest for davidusb/image-aarch64-emhass:0.2.27 not found: manifest unknown: manifest unknown")
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/hassio/update.py", line 167, in async_install
    await async_update_addon(self.hass, slug=self._addon_slug, backup=backup)
  File "/usr/src/homeassistant/homeassistant/components/hassio/handler.py", line 51, in _wrapper
    raise HassioAPIError(data["message"])
homeassistant.components.hassio.handler.HassioAPIError: Can't install davidusb/image-aarch64-emhass:0.2.27: 404 Client Error for http+docker://localhost/v1.41/images/create?tag=0.2.27&fromImage=davidusb%2Fimage-aarch64-emhass&platform=linux%2Farm64: Not Found ("manifest for davidusb/image-aarch64-emhass:0.2.27 not found: manifest unknown: manifest unknown")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 200, in handle_call_service
    await hass.services.async_call(
  File "/usr/src/homeassistant/homeassistant/core.py", line 1755, in async_call
    task.result()
  File "/usr/src/homeassistant/homeassistant/core.py", line 1792, in _execute_service
    await cast(Callable[[ServiceCall], Awaitable[None]], handler.job.target)(
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 213, in handle_service
    await service.entity_service_call(
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 678, in entity_service_call
    future.result()  # pop exception if have
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 958, in async_request_call
    await coro
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 715, in _handle_entity_call
    await result
  File "/usr/src/homeassistant/homeassistant/components/update/__init__.py", line 154, in async_install
    await entity.async_install_with_progress(version, backup)
  File "/usr/src/homeassistant/homeassistant/components/update/__init__.py", line 413, in async_install_with_progress
    await self.async_install(version, backup)
  File "/usr/src/homeassistant/homeassistant/components/hassio/update.py", line 169, in async_install
    raise HomeAssistantError(f"Error updating {self.title}: {err}") from err
homeassistant.exceptions.HomeAssistantError: Error updating EMHASS: Can't install davidusb/image-aarch64-emhass:0.2.27: 404 Client Error for http+docker://localhost/v1.41/images/create?tag=0.2.27&fromImage=davidusb%2Fimage-aarch64-emhass&platform=linux%2Farm64: Not Found ("manifest for davidusb/image-aarch64-emhass:0.2.27 not found: manifest unknown: manifest unknown")

[davidusb-geek]

Yes you have to wait just one hour approx., the images are being built.

[lovekull76]

Tested and....

Optim + publish + restart add-on + publish = fail:

[2023-01-30 20:26:08,208] INFO in command_line: Setting up needed data [2023-01-30 20:26:08,250] INFO in web_server: >> Publishing data... [2023-01-30 20:26:08,250] INFO in command_line: Publishing data to HASS instance [2023-01-30 20:26:08,250] ERROR in command_line: File not found error, run an optimization task first. [2023-01-30 20:26:08,250] ERROR in app: Exception on /action/publish-data [POST] Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2525, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1822, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1820, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1796, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 139, in actioncall = publish_data(input_data_dict, app.logger) File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 288, in publish_data idx_closest = opt_res_latest.index.get_indexer([now_precise], method='nearest')[0] UnboundLocalError: local variable 'opt_res_latest' referenced before assignment

I need to do optim again after reload of add-on before publish :( The website is showing the updated optim data but can't do publish until I do a new optim.

Have not tried to reload HA yet.

Best regards, L

[lovekull76]

PS.

Reload of HA works ok, I think. I get this after HA reload + publish (no new optim):

[2023-01-30 20:55:35,526] INFO in command_line: Setting up needed data [2023-01-30 20:55:35,530] INFO in web_server: >> Publishing data... [2023-01-30 20:55:35,530] INFO in command_line: Publishing data to HASS instance [2023-01-30 20:55:35,663] INFO in retrieve_hass: The received response code is not recognized [2023-01-30 20:55:35,705] INFO in retrieve_hass: The received response code is not recognized [2023-01-30 20:55:35,754] INFO in retrieve_hass: The received response code is not recognized [2023-01-30 20:55:35,793] INFO in retrieve_hass: The received response code is not recognized [2023-01-30 20:55:35,839] INFO in retrieve_hass: The received response code is not recognized [2023-01-30 20:55:35,887] INFO in retrieve_hass: The received response code is not recognized [2023-01-30 20:55:35,913] INFO in retrieve_hass: The received response code is not recognized

But I think the publish worked.

//L

[lovekull76]

PS2.

I don't have a CSV in my share folder. I do have injection_dict.pkl and params.pkl.

I'm trying to backtrack were the CSV is saved in your code but I'm not there yet :) I can really appreciate your work now that I'm poking around in the code!

What strikes me is that the error references line 288 in command_line.py:

File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 288, in publish_data idx_closest = opt_res_latest.index.get_indexer([now_precise], method='nearest')[0]

If I check this line in the repo that is not the same line. Could my install of EMHASS been messed up and I'm not running the 0.2.27 version? The Info page says so anyway. EDIT: And now I see that there was changes done to this file 41 minutes ago :) Do I re-download to get the updated version?

[davidusb-geek]

Thanks for the tests and sorry for these bugs. Yes now I've just treated some more bugs regarding these paths. They should be solved now (I hope for good). Tested on add-on restart, publish and HA restart. It is working now and I'm able to find the CSV in the share folder.

Just wait until the images are built again (~1h) and then update to the latest version.

[lovekull76]

Great work! I'll check it out tomorrow. I don't want to do a new optim right now. My battery is charging as we speak and the price is high tomorrow so I don't want to mess up my current optim :) I really appreciate your work on this!

[lovekull76]

I think it works now. I have not had any problems this far anyway. Great work!

[davidusb-geek]

Thanks. Yes I've tested it in all possible ways this week and everything is fine, the data is always retained in the share folder. Closing this as completed