FR: Return forecast values to home assistant sensor entity attributes

[purcell-lab]

A feature request.

The forecast values are currently written to the .csv file within the file system, but are unable to be accessed by home assistant.

Would it be possible to include the forecasts as attributes of the sensor entities published to home assistant.

This would enable the creation of the same chart of values as the web UI, but in Lovelace using the apexcharts-card

For example the SolCast integration posts forecast PV through a series of sensors and attributes.

My energy provider also provides future price forecasts in entity attributes:

When visualized in apexcharts looks like this:

[davidusb-geek]

Yes, I don't see any problem with doing this. I'll just need to look how to do it but I guess that it can't be hard to pass a list of values as an attribute to a sensor. I'll look into that

[davidusb-geek]

Hi, I just published a new version where this was added. Test it to see if it fits your needs.

[davidusb-geek]

I tried this block of code:

            forecast_list = copy.deepcopy(data_df).loc[data_df.index[idx]:].reset_index()
            forecast_list.columns = ['timestamps', entity_id]
            ts_list = [str(i) for i in forecast_list['timestamps'].tolist()]
            vals_list = [np.round(i) for i in forecast_list[entity_id].tolist()]
            forecast_list = [{'timestamp': ts_list[i], entity_id.split('sensor.')[1]: vals_list[i]} for i in range(len(ts_list))]

But this is the result, not quite what a typical forecast sensor shows:

[davidusb-geek]

Ok just found how they are doing it in Amber: https://github.com/home-assistant/core/blob/dev/homeassistant/components/amberelectric/sensor.py

I will update that for the next release...

[purcell-lab]

Great, I can see the values there now, so can do something with them.

I would be good if you can get into a similar structure as amber integration as that is quite easy to work with using map(attribute):

post_mpc_optim: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"prod_price_forecast\":{{(
          (state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list)) 
          }},\"load_cost_forecast\":{{(
          (state_attr('sensor.amber_general_forecast', 'forecasts') |map(attribute='per_kwh')|list))
          }},\"prediction_horizon\":{{(
          (state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list|length)) 
          }},\"soc_init\":{{states('sensor.powerwall_charge')|float(0)/100}},\"soc_final\":0.01,\"def_total_hours\":[4,5,3,1]}' http://localhost:5000/action/naive-mpc-optim"

post_mpc_optim: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"prod_price_forecast\":[0.28, 0.33, 0.37, 0.32, 0.31, 0.31, 0.28, 0.27, 0.37, 0.46, 0.46, 0.46, 0.46, 0.46, 0.33, 0.33, 0.46, 0.46, 0.46, 0.37, 0.33, 0.33, 0.32, 0.3, 0.3, 0.31, 0.46, 0.33, 0.31, 0.3, 0.31, 0.3, 0.31, 0.3, 0.3, 0.26, 0.26, 0.26],\"load_cost_forecast\":[0.4, 0.45, 0.51, 0.45, 0.44, 0.44, 0.4, 0.4, 0.51, 0.61, 0.61, 0.61, 0.61, 0.61, 0.46, 0.46, 0.6, 0.61, 0.61, 0.5, 0.46, 0.45, 0.45, 0.43, 0.43, 0.44, 0.61, 0.46, 0.44, 0.42, 0.44, 0.42, 0.43, 0.43, 0.42, 0.38, 0.38, 0.38],\"prediction_horizon\":38,\"soc_init\":0.85,\"soc_final\":0.01,\"def_total_hours\":[4,5,3,1]}' http://localhost:5000/action/naive-mpc-optim"

[davidusb-geek]

Yes I will update this for the next release. That's a nice implementation of the shell command using templates and mapping the attributes. Do you mind if I borrow your code to put it as an example in the docs? It could help people with similar integrations.

A side comment on the values that you're passing for soc_final and def_total_hours. For the SOC if you do this at each iteration of the MPC it will always look to empty the battery. I think that you may lose optimality doing this. An idea could be to run the dayahead optimization just once at the beginning of a new day and set the battery optimal power and SOC trajectories for the day. Then launch the MPC iteratively at a relative high frequency (10min?) and fix that final SOC to match the optimal trajectory for whole day computed with the dayahead optimization. The MPC will still let you optimize your system on short term uncertainties, but you will keep an optimal goal for your complete day. As for the def_total_hours what I'm doing is that I defined sensors for my deferrable loads that give me the current time in hours that those loads have done on the current day. So that when I perform the MPC optimization I'm passing the difference of the daily goal and the total hours ran for each deferrable load. Be careful because it doesn't make much sense to keep def_total_hours at fixed values as shown in your code snippet.

Below is the code I use to compute running hours sensor for one of my deferrable loads, the water heater:

  - platform: history_stats
    name: Water Heater running today
    entity_id: sensor.water_heater_status
    state: 'Running'
    type: time
    start: '{{ now().replace(hour=0).replace(minute=0).replace(second=0) }}'
    end: '{{ now() }}'
  - platform: template
    sensors:
      water_heater_status:
        value_template: '{% if is_state("switch.water_heater_switch", "on") %}Running{% else %}Stopped{% endif %}'
        friendly_name: 'Water Heater Status'
        icon_template: mdi:play-pause
      water_heater_time_on:
        value_template: '{{ states("sensor.water_heater_running_today") | float }}'
        friendly_name: 'Water Heater Hours Running'
        icon_template: mdi:timer

[purcell-lab]

Documentation for examples is going to be very important. I have already had a few requests to publish my configuration as others are wanting to utilise the benefits of emhass. Things are still moving quite quickly with your development so maybe the GitHub wiki could be a good place to document different configurations.

I did contemplate what values to use for soc_final as it very much drives behaviour at the end of the cycle. If it is too high it will artificially charge or as you say too low and it will fully discharge. My observations are that it only really effects the final set of time periods in an optimisation.

I chose the minimum for soc_final as a fully discharged battery at the end of the cycle does represent the maximum achievable total value of the cost function. Additionally as I'm running MPC optimisation every 5 minutes the actual effects of the soc_final are never actually realised in the posted values for the controlling functions, for my forecasts this is always 30-48 time slots in the future.

I understand this makes the actual run hours not deterministic and have been using this sliding window approach both with emhass and my prior rules based approach. I have observed that if you have a bizzare set of forecasts that continually increase or decrease between optimisations you can end up running for more or less than the desired hours. Given some of the other estimations in the system if I get an extra or hour or less for pool pump or EV charging it isn't a disaster.

I also have some working formula for the run time if deferrable loads, but I wanted to get the basics running first.

    amber_lowest_cost:
      friendly_name: "Lowest Cost 90 mins"
      unit_of_measurement: $/kWh
      value_template: "{{ (state_attr('sensor.amber_general_forecast', 'forecasts') | map(attribute='per_kwh')|list|sort).4}}"

    amber_highest_price:
      friendly_name: "Highest Price 90 minutes"
      unit_of_measurement: $/kWh
      value_template: "{{ (state_attr('sensor.amber_feed_in_forecast', 'forecasts') | map(attribute='per_kwh')|list|sort(reverse=true)).4}}"

    ev_charging_30mins:
      value_template : "{{(((90-states('sensor.duka_battery_sensor')|int(0)) / 100 * 8* 2)|int(0))}}"

[davidusb-geek]

Yes thanks for your input. I need to work the documentation with better explanation, graphics and more practical examples. Yes the wiki can be an option as well as readthedocs as it is published automatically ;-)

I see the thing with your MPC implementation is that you're using a prediction_horizon that is always the size of your available forecast list. So in this case it makes sense to fix those values and the whole is valid on a sliding window. But this is as doing a dayahead optimization repeatedly every 5mins. You can still use it like this I think is totally valid, but the main goal of the MPC is to tame variability and uncertainties at short frequency and with shorter prediction_horizon. This may give you the opportunity to reduced your optimization time step and have a gain in precision. This could be interesting specially for the load consumption forecast. For now this forecast is just a naive persistence approach in emhass. The goal is to improve this in the near future to provide better load consumption forecast. Then the MPC could use more precise short term load consumption forecast to optimize at higher frequencies. Again I need to work the docs to explain all this... It is in progress...

[purcell-lab]

Lots of possibilities.

I am envisioning a MPC optimisation called every 5 minutes (or less) with the first method_ts_round with actual current values for PV, load, buy & sell and forecasts for dayahead optimisation. timestep stays at 30 minutes as I cannot get finer resolution for my buy/ sell.

The desired effect is if the sun goes behind a big cloud, load jumps up or down unexpectedly or price/ cost change radically in the 5 minutes. It maybe optimal to change battery or deferrable load activity. Of course you don't want to switch these large loads to quickly otherwise relays and the like will burn out. But it may make sense to switch loads inside the 30 minute window based off actual measured values.

Similarly for variable (semi continuous) loads, my battery will match excess solar curves for charging and I have scripts to match EV charging to excess solar. But for those to be > 85% effective you need minute or second resolution. My battery gives me second resolution to match solar/ load to +/- 5 kW and my EV charging gives me minute resolution for 0-12 kW. So the big deferrable loads (water heating, HVAC, pool pump) only need to be in the ball park -5 kW to +17 kW to still be optimal, as the batteries can be the shock absorber for external variations.

[purcell-lab]

Nice, I can visualize this very nicely within lovelace and it updates in 'real' time.

type: custom:apexcharts-card
experimental:
  color_threshold: true
graph_span: 24h
span:
  start: minute
header:
  show: true
  title: EMHASS Forecasts
  show_states: false
  colorize_states: true
series:
  - entity: sensor.p_pv_forecast
    name: pv
    data_generator: |
      return entity.attributes.forecasts.map((entry) => {
        return [new Date(entry.date), entry.p_pv_forecast];
      });
  - entity: sensor.p_batt_forecast
    name: p_batt
    data_generator: |
      return entity.attributes.forecasts.map((entry) => {
        return [new Date(entry.date), entry.p_batt_forecast];
      });
  - entity: sensor.p_load_forecast
    name: p_load
    data_generator: |
      return entity.attributes.forecasts.map((entry) => {
        return [new Date(entry.date), entry.p_load_forecast];
      });
  - entity: sensor.p_deferrable0
    name: p_deferable0
    data_generator: |
      return entity.attributes.deferrables_schedule.map((entry) => {
        return [new Date(entry.date), entry.p_deferrable0];
      });
  - entity: sensor.p_deferrable1
    name: p_deferable1
    data_generator: |
      return entity.attributes.deferrables_schedule.map((entry) => {
        return [new Date(entry.date), entry.p_deferrable1];
      });
yaxis:
  - apex_config:
      forceNiceScale: true

[davidusb-geek]

Nice, very cool visualizations...