search

davidusb-geek / emhass-add-on

The Home Assistant Add-on for EMHASS: Energy Management Optimization for Home Assistant
MIT License
96 stars 29 forks source link

Issue loading in EMHASS Nordpool energy prices #36

Closed ghost closed 1 year ago

ghost commented 1 year ago

Hello,

I'm very interested in the EMHASS Add-on for efficiently charging my home-batteries via my PV-System en when the energy prices are low directly from the Grid. Also I want to sell excessive energy back to the grid to make the ROI of my Battery system as short as possible.

I have read the EMHASS documentation and installed the EMHASS add-on. I’m using a dedicated NUC running HASSIO. I also have installed the Nordpool integration showing the RAW-Data for today and tomorrow. I renamed this as suggested in “nordpool”. I entered the curl commands in my configuration.yaml file as described. When starting up HA the EMHASS log is showing as follows:

  s6-rc: info: service s6rc-oneshot-runner: starting
  s6-rc: info: service s6rc-oneshot-runner successfully started
  s6-rc: info: service fix-attrs: starting
  s6-rc: info: service fix-attrs successfully started
  s6-rc: info: service legacy-cont-init: starting
  s6-rc: info: service legacy-cont-init successfully started
  s6-rc: info: service legacy-services: starting
  services-up: info: copying legacy longrun emhass (no readiness notification)
  s6-rc: info: service legacy-services successfully started
  [2023-02-28 18:30:29,180] INFO in web_server: Launching the emhass webserver at: http://0.0.0.0:5000
  [2023-02-28 18:30:29,181] INFO in web_server: Home Assistant data fetch will be performed using url: http://supervisor/core/api
  [2023-02-28 18:30:29,181] INFO in web_server: The data path is: /share
  [2023-02-28 18:30:29,183] INFO in web_server: Using core emhass version: 0.3.36
  [2023-02-28 18:32:12,574] INFO in web_server: EMHASS server online, serving index.html...
  [2023-02-28 18:32:27,495] INFO in command_line: Setting up needed data
  [2023-02-28 18:32:27,614] INFO in forecast: Retrieving weather forecast data using method = scrapper
  [2023-02-28 18:32:30,176] INFO in forecast: Retrieving data from hass for load forecast using method = naive
  [2023-02-28 18:32:30,177] INFO in retrieve_hass: Retrieve hass get data method initiated...
  [2023-02-28 18:32:30,797] INFO in web_server: EMHASS server online, serving index.html...
  [2023-02-28 18:32:32,487] INFO in web_server:  >> Performing dayahead optimization...
  [2023-02-28 18:32:32,487] INFO in command_line: Performing day-ahead forecast optimization
  [2023-02-28 18:32:32,494] INFO in optimization: Perform optimization for the day-ahead
  [2023-02-28 18:32:32,738] INFO in optimization: Status: Optimal
  [2023-02-28 18:32:32,739] INFO in optimization: Total value of the Cost function = -16.5
  [2023-02-28 18:53:28,100] INFO in web_server: EMHASS server online, serving index.html...
  [2023-02-28 18:54:09,069] INFO in web_server: EMHASS server online, serving index.html...
  [2023-02-28 18:54:18,048] INFO in command_line: Setting up needed data
  [2023-02-28 18:54:18,050] INFO in forecast: Retrieving weather forecast data using method = scrapper
  [2023-02-28 18:54:19,274] INFO in forecast: Retrieving data from hass for load forecast using method = naive
  [2023-02-28 18:54:19,275] INFO in retrieve_hass: Retrieve hass get data method initiated...
  [2023-02-28 18:54:20,616] INFO in web_server:  >> Performing dayahead optimization...
  [2023-02-28 18:54:20,616] INFO in command_line: Performing day-ahead forecast optimization
  [2023-02-28 18:54:20,623] INFO in optimization: Perform optimization for the day-ahead
  [2023-02-28 18:54:20,748] INFO in optimization: Status: Optimal
  [2023-02-28 18:54:20,749] INFO in optimization: Total value of the Cost function = -16.5
  [2023-02-28 18:54:39,450] INFO in web_server: EMHASS server online, serving index.html...

When running the EMHASS UI I get an updated optimization proposal graph and table but NOT with the energy prices as listed in the nordpool RAW-Data.

index | P_PV | P_Load | P_deferrable0 | P_deferrable1 | P_deferrable2 | P_deferrable3 | P_grid_pos | P_grid_neg | P_grid | unit_load_cost | unit_prod_price | cost_profit | cost_fun_profit -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- 2023-02-28 19:00:00+01:00 | -1.500000 | 3.033833 | 0.0 | 0.0 | 0.0 | 7000.0 | 7004.533800 | 0.000000 | 7004.533800 | 0.1419 | 0.065 | -0.496972 | -0.496972 2023-02-28 19:30:00+01:00 | -1.500000 | 2.812722 | 0.0 | 1000.0 | 0.0 | 7000.0 | 8004.312700 | 0.000000 | 8004.312700 | 0.1419 | 0.065 | -0.567906 | -0.567906 2023-02-28 20:00:00+01:00 | -1.500000 | 1.527500 | 0.0 | 0.0 | 5000.0 | 0.0 | 5003.027500 | 0.000000 | 5003.027500 | 0.1419 | 0.065 | -0.354965 | -0.354965 2023-02-28 20:30:00+01:00 | -1.500000 | 1.086444 | 0.0 | 1000.0 | 5000.0 | 0.0 | 6002.586400 | 0.000000 | 6002.586400 | 0.1419 | 0.065 | -0.425884 | -0.425884 2023-02-28 21:00:00+01:00 | -1.500000 | 1.440833 | 0.0 | 0.0 | 0.0 | 7000.0 | 7002.940800 | 0.000000 | 7002.940800 | 0.1419 | 0.065 | -0.496859 | -0.496859 2023-02-28 21:30:00+01:00 | -1.500000 | 1.079778 | 0.0 | 0.0 | 0.0 | 7000.0 | 7002.579800 | 0.000000 | 7002.579800 | 0.1419 | 0.065 | -0.496833 | -0.496833 2023-02-28 22:00:00+01:00 | -1.500000 | 3.068222 | 0.0 | 0.0 | 0.0 | 7000.0 | 7004.568200 | 0.000000 | 7004.568200 | 0.1419 | 0.065 | -0.496974 | -0.496974 2023-02-28 22:30:00+01:00 | -1.500000 | 2.724111 | 0.0 | 0.0 | 5000.0 | 0.0 | 5004.224100 | 0.000000 | 5004.224100 | 0.1419 | 0.065 | -0.355050 | -0.355050

I probably miss something simples but after looking for many hours I hope that somebody could help me out here.

I just started using Home assistant so Im not so experienced yet in HA so any guidance is more then welcome.

Best regards,

Pieter

davidusb-geek commented 1 year ago

Hi Peter, great this is one of the reasons why emhass exists, to help minimize roi's. In your case as the optimization is already running with no problem your issue will be with the curl command. Could you please share the exact command that you are using? And did you check that you have correct data on your prices list from Nordpool?

ghost commented 1 year ago

Hi David,

Thanks for your reply. I use in my configuration.yaml the following curl command for the nordpool integration.

shell_command: trigger_nordpool_forecast: 'curl -i -H "Content-Type: application/json" -X POST -d ''{"load_cost_forecast":{{((state_attr(''sensor.nordpool'', ''raw_today'') | map(attribute=''value'') | list + state_attr(''sensor.nordpool'', ''raw_tomorrow'') | map(attribute=''value'') | list))[now().hour:][:24] }},"prod_price_forecast":{{((state_attr(''sensor.nordpool'', ''raw_today'') | map(attribute=''value'') | list + state_attr(''sensor.nordpool'', ''raw_tomorrow'') | map(attribute=''value'') | list))[now().hour:][:24]}}}'' http://localhost:5000/action/dayahead-optim' dayahead_optim: 'curl -i -H "Content-Type:application/json" -X POST -d ''{}'' http://localhost:5000/action/dayahead-optim'

I renamed the original name of the nordpool sensor (sensor.nordpool_kwh_nl_eur_3_10_021) into nordpool as suggested. The data the sensor is showing:

`state_class: measurement average: 0.17475 off_peak_1: 0.16575 off_peak_2: 0.184 peak: 0.17766666666666667 min: 0.141 max: 0.235 mean: 0.1585 unit: kWh currency: EUR country: Netherlands region: NL low_price: false price_percent_to_average: 1.3333333333333335 today:

image

I hope the above helps determining what the issue is on my side.

Br Pieter

davidusb-geek commented 1 year ago

Ok two more things.

First, try to use this same syntax:

shell_command:
  trigger_nordpool_forecast: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list  + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }},\"prod_price_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list  + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24]}}}' http://localhost:5000/action/dayahead-optim"

Mostly for the load_cost_forecast between \".

So in your case the load forecast part should look like:

\"load_cost_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list  + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }}

And secondly, if you put this in Developer tools / Template what is the result?:

{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list  + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }}
ghost commented 1 year ago

I have removed the old curl command and replaced it by the one you suggested above. After restarting HA, starting EMHASS and running Day-Ahead Optmimation and refresh te cash of my browser I still see the standard prices as listed in the EMHASS config file.

When running the given string in Dev.Tools / Template I get the folowing result:

image

Any more tips / suggestions?

davidusb-geek commented 1 year ago

Sorry I just updated my last comment with the correct template syntax. Try that please. I'm guessing that the moment when you get a correct list of values in DevTools/Template then it will work correctly on EMHASS.

ghost commented 1 year ago

I'm getting closer. When entering the given string in Dev/Tools/Template I get the following which is, I guess, not complete as I only see prices for tomorow.

image

When adding the line in configuration.yaml I had to add " at the end of the curl command to get no error when checking the configuration.yaml file.

Running EMHASS still does not give me the correct energy prices. Probebly becasue the generated data is not compleet.

davidusb-geek commented 1 year ago

Yes, that is the cause. Your list needs to have the correct size. Using that template editor check the raw_today data only, there is no data there? That template should correctly concatenate those today + tomorrow values

ghost commented 1 year ago

When I eneter the shortened line for only RAW-Today such as:

{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list ))}}

The i see the following output:

image

Which are the prices of today. I notice now that this list starts exactly with the first price listed in RAW-sensor Data. When using the full string you gave me it see in the first lines some difference. Looks like the list is a mix of the prices.

I will take a closer look and play arround a bit more tomorow. I know now where to look.

Thanks for your help till so far.

Br Pieter

ghost commented 1 year ago

I just quickly tried the prices tommorow in the dev/tool template.

{{((state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }}

Then the outcome is only one vallue.

image

ghost commented 1 year ago

I have played a bit around with the code and found out that when I remove the format statement [now().hour:[]:24] at the end of the code and test this in the Dev/Tools/Template it is giving me the prices for today and tomorrow, so 24 values.

I have now used the following code:

{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))}}

With as result:

image

When adding the the formatting statement [now().hour:[]:24] I get 24 values listed; 7 values for today for the rest of the hours in this day and 17 for tomorrow, in total 24 values/prices. I checked this under states and the values listed are correct.

When I change the value 24 to 48 [now().hour:[]:48], then I see in Dev/tools/template all prices from now till the end of tomorrow.

When I put in configuration.yaml the full curll-string

trigger_nordpool_forecast: 'curl -i -H "Content-Type: application/json" -X POST -d ''{"load_cost_forecast":{{((state_attr(''sensor.nordpool'', ''raw_today'') | map(attribute=''value'') | list + state_attr(''sensor.nordpool'', ''raw_tomorrow'') | map(attribute=''value'') | list))[now().hour:][:24] }},"prod_price_forecast":{{((state_attr(''sensor.nordpool'', ''raw_today'') | map(attribute=''value'') | list + state_attr(''sensor.nordpool'', ''raw_tomorrow'') | map(attribute=''value'') | list))[now().hour:][:24]}}}'' http://localhost:5000/action/dayahead-optim'

then I still do not get the prices in the EMHASS table after running the optimization task and a refresh of my browser cash. Also not when I remove the [now().hour:[]:24] statement, or change the value from 24 to 48 or completely remove this.

Any further thoughts how I can solve this here. Would it be an option to generate instead a CSV-File?

I'm looking forward to any helpfull comments/suggestions.

Br. Pieter

davidusb-geek commented 1 year ago

It depends on the action that you want to execute and the time step that you are using from the add-on configuration pane. So if you have a 30 min time step and you want to do a dayahead-optim as in your code snippet, then emhass will need a list of 48 values. Other users had this same problem with Nordpool, check the community forum post for emhass for solutions. A possible solution if during the day the length of that list is not always enough for emhass then you can may be switch to use MPC. For this you have to change your endpoint name and pass some additionnal parameters, notably the prediction horizon. So for example if during the day your list have at least 24 items at any given time, the do MPC with a 12 hours prediction horizon. Check the documentation, there are examples on how to set MPC.

ghost commented 1 year ago

Thanks for your input. I will take a look.

overas commented 1 year ago

@PieterD67, I am using Nordpool data myself. You can look at my setup here: https://github.com/davidusb-geek/emhass-add-on/issues/37

ghost commented 1 year ago

Thanks for sharing the info. Seems you have the same kind of issue as I'm seeing. The Nordpool data is retreived OK but not used by during the optimasation task in EMHASS. I understood that David has everything running within a docker based system. Maybe this is a pure HA-OS issue. I hope too have some time this weekend to take a closer look.

davidusb-geek commented 1 year ago

I will look into this. And HA-OS is a pure docker based system. There has to be a bug somewhere, it will be fixed.

davidusb-geek commented 1 year ago

Yesterday I tried to debug this. I wrote a new special test to try to reproduce the issue but with no success. In my tests the passed list of values is always correctly integrated into the optimization. I'll keep looking.

davidusb-geek commented 1 year ago

After all the changes that you have made since creating this issue, could you please restart the add-on then don't do any data publish task, then show the complete logs of just running the day-ahead optimization task? Also please share your add-on config.yaml. Hiding any possible sensible information.

davidusb-geek commented 1 year ago

https://github.com/davidusb-geek/emhass-add-on/issues/37#issuecomment-1454716655