Add grid import and export sensors for use by HA energy dashboard

[gmcmicken]

Home assistant's energy dashboard is expecting net energy imported and exported, but this component only gives total energy consumed and produced and is therefore incompatible with this feature.

Using the lifetime energy production and consumption values, we can determine a delta between current and last poll value, and then determine if we imported or exported this amount of energy to/from the grid.

This PR is proof of concept only. I am not a python developer and the method I am using to retrieve the last sensor value is likely not correct (as I believe entity IDs can be changed). I'm hoping you know how to pull the last value from the DataUpdateCoordinator or elsewhere, and replace those four lines.

[bouyssic]

I couldn't make half of it myself! #Respect

Following this if you need testers

[gmcmicken]

I couldn't make half of it myself! #Respect

Following this if you need testers

You can test it right now if you want. Delete the integration, go to HACS, remove it there and delete the custom repository entry. Add back custom repository "_https://github.com/gmcmicken/home_assistant_custom_envoy_" and then add the integration to HA again. Just a warning though, don't rename any of your envoy entities! The code relies on the entity name to be as it was set initially.

[bouyssic]

Nope unfortunately it doesn't work

I'm sure I updated your repo as I can see your changes in custom_components

[gmcmicken]

Nope unfortunately it doesn't work

I'm sure I updated your repo as I can see your changes in custom_components

Sorry about that, I missed a line in this PR that I had in my running copy! You can try it again if you like.

[bouyssic]

No need to be sorry! Checking that right now

[bouyssic]

It's working, but showing 0 for now. I'll leave it running some days

[gmcmicken]

It's working, but showing 0 for now. I'll leave it running some days

The HA energy dashboard needs up to two hours to start drawing a graph (one hour data on the hour). You can check your grid import history before then by going to entities and looking at the graph in the more info pop-up

[bouyssic]

Yeah I noticed afterwards.

I've been restarting HA a lot this last couple of hours so we'll see with time.

Thanks for all the help!

[toadkicker]

@gmcmicken pulled your branch and tested against my setup and its working great. Thanks for the contrib.

[gmcmicken]

@gmcmicken pulled your branch and tested against my setup and its working great. Thanks for the contrib.

Great, yea mine has been good so far as well.

[toadkicker]

@bouyssic are you still showing 0's ? Can you share your dashboard setup?

[bouyssic]

Nope all good mate, sorry I forgot to confirm.

I has been working like a charm

[briancmpbll]

These calculations assume no storage. For a system with batteries you will have to subtract the energy charged and discharged from the energy exported and imported, respectively.

I tried for a couple hours today to get this working, but I need to account both for systems with and without batteries and that is complicating things. I don't know if this sort of complicated calculation is something I want to put in this integration.

If you can account for this in your PR, then cool, but you'll need to work with the BatteryEnergyChangeEntities for energy charged and discharged in sensor.yml. I think a proper solution would conditionally add coordinated entities for systems without batteries or entities that use async_track_state_change_event to track their respective battery entities and update when they change.

[gmcmicken]

Hmm I didn't think about battery storage! Do you think that is a sizable amount of users? Could we get away with doing grid import/export conditionally only for users without batteries?

I might take a stab at re-writing the grid import/export inside sensor.py as a new EnvoyGridEntity and tie that to a state change on the lifetime energy entities. But again I have no experience with the hass API so it will take me considerably longer than someone who knows what they're doing.

[toadkicker]

Seeing 0's for these values after pulling latest. Note I don't have batteries.

[bouyssic]

Seeing 0's for these values after pulling latest. Note I don't have batteries.

Hey @toadkicker ,

For me it took a couple of hours before seing everything working. How long since you installed?

Chris

[toadkicker]

Pulled yesterday morning so its been 24 hours.

[gmcmicken]

Pulled yesterday morning so its been 24 hours.

Can you restart HA then pull the logs after 5 min? We'll see what the issue was from that.

[toadkicker]

These are the right entities?

[gmcmicken]

These are the right entities?

Almost. There should be grid import and grid export for your energy dashboard.

But the problem is actually that your energy production totals are 0. Therefore the grid cannot be calculated. Either your unit is reporting 0 or the API call isn't working to retrieve them.

[samyun]

@briancmpbll what needs to be done to check this in?

[gmcmicken]

I noticed the JSON result seems to contain net-consumption now, and for anyone who is interested and has a v7+ device, the auth token lasts 1 year now so you can just add a few lines to your HA configuration and ditch this whole integration altogether (using "RESTful" and "Riemann sum integral" sensors).

Consumption
    type: eim
    activeCount: 1
    measurementType: total-consumption
    readingTime: 1678253537
    wNow: 754.267
    whLifetime: 7550725.811
    varhLeadLifetime: 1098099.27
    varhLagLifetime: 1241378.569
    vahLifetime: 7418505.635
    rmsCurrent: 12.483
    rmsVoltage: 240.131
    reactPwr: -1033.872
    apprntPwr: 2997.587
    pwrFactor: 0.25
    whToday: 54330.811
    whLastSevenDays: 499998.811
    vahToday: 49846.635
    varhLeadToday: 6904.27
    varhLagToday: 7920.569

    type: eim
    activeCount: 1
    measurementType: net-consumption
    readingTime: 1678253537
    wNow: 754.267
    whLifetime: 6697333.955
    varhLeadLifetime: 1093101.675
    varhLagLifetime: 135537.447
    vahLifetime: 7418505.635
    rmsCurrent: 8.64
    rmsVoltage: 240.231
    reactPwr: -578.217
    apprntPwr: 1037.206
    pwrFactor: 0.73
    whToday: 0
    whLastSevenDays: 0
    vahToday: 0
    varhLeadToday: 0
    varhLagToday: 0

[DigitalBites]

I manually applied the above changes and it is all working for me in HA along with the energy dashboard. There are some differences that I see from 'enlighten' vs HA, which generally is not too significant but it is working. How do we either get this Pull request either improved or included?

[gmcmicken]

I don't think it needs to be anymore, if @briancmpbll can add the new net-consumption values from the json API it would done in a jiffy.

[briancmpbll]

I took a look at this today and it looks like I can add sensors for the wNow value in the json, but the wh values don't seem usable. whToday and whLastSevenDays are always zero and whLifetime looks like it's strictly increasing and does not decrease when there is negative net consumption (when you're producing more than you're consuming). Users would likely need to add their own Reimann sum integral sensors to track net consumption and production.

I don't have any more time to work on it this weekend, so if someone else is able to make these changes "in a jiffy", please feel free 😃

Whoever implements this will need to add a net_consumption function to envoy reader that does the same thing as async def consumption(self): on line 540, but with raw_json["consumption"][1]["wNow"] instead of [0]. You will also need to add entities similar to BatteryEnergyChangeEntity that give you net consumption and net production depending on the sign of the net consumption value.

[gmcmicken]

I took a look at this today and it looks like I can add sensors for the wNow value in the json, but the wh values don't seem usable. whToday and whLastSevenDays are always zero and whLifetime looks like it's strictly increasing and does not decrease when there is negative net consumption (when you're producing more than you're consuming). Users would likely need to add their own Reimann sum integral sensors to track net consumption and production.

I don't have any more time to work on it this weekend, so if someone else is able to make these changes "in a jiffy", please feel free 😃

Whoever implements this will need to add a net_consumption function to envoy reader that does the same thing as async def consumption(self): on line 540, but with raw_json["consumption"][1]["wNow"] instead of [0]. You will also need to add entities similar to BatteryEnergyChangeEntity that give you net consumption and net production depending on the sign of the net consumption value.

I feel bad leaving you guys with this alone but I'm not using the integration anymore and I find the result is better with the rest and integral sensor so I don't really want to go back anyway.

[briancmpbll]

PR abandoned. Looks like #55 might add this functionality. Need to take a look.

[danepowell]

55 looks to add power import/export, but not energy import/export totals. 😢

What are other people doing to get total grid energy imports/exports right now? @gmcmicken can you elaborate on why this approach is bad?

[gmcmicken]

The current problem is that the net consumption data from enphase is power only, the energy portion is not correct. Instead of using the integration at all, I use the following config and it's been rock solid for me.

sensor:
  - platform: rest
    name: "Enphase"
    unique_id: enphase
    resource_template: https://[enphase ip]/production.json
    scan_interval: 15
    verify_ssl: false
    headers:
      Content-Type: application/json
      Authorization: >
        Bearer [your token]
    json_attributes:
      - production
      - consumption
    value_template: "OK"
  - platform: integration
    name: "Enphase Produced"
    unique_id: enphase_produced
    source: sensor.enphase_production
    unit_prefix: k
    method: left
    round: 2
  - platform: integration
    name: "Enphase Consumed"
    unique_id: enphase_consumed
    source: sensor.enphase_consumption
    unit_prefix: k
    method: left
    round: 2
  - platform: integration
    name: "Enphase Grid Imported"
    unique_id: enphase_grid_imported
    source: sensor.enphase_grid_import
    unit_prefix: k
    method: left
    round: 2
  - platform: integration
    name: "Enphase Grid Exported"
    unique_id: enphase_grid_exported
    source: sensor.enphase_grid_export
    unit_prefix: k
    method: left
    round: 2

template:
  - sensor:
      - name: "Enphase Production"
        unique_id: enphase_production
        state: "{{ state_attr('sensor.enphase', 'production')[1]['wNow'] | round }}"
        device_class: power
        unit_of_measurement: W
      - name: "Enphase Consumption"
        unique_id: enphase_consumption
        state: "{{ state_attr('sensor.enphase', 'consumption')[0]['wNow'] | round }}"
        device_class: power
        unit_of_measurement: W
      - name: "Enphase Grid Import"
        unique_id: enphase_grid_import
        state_class: measurement
        unit_of_measurement: W
        device_class: power
        state: >
            {{ [ 0, state_attr('sensor.enphase', 'consumption')[1]['wNow'] ] | max }}
      - name: "Enphase Grid Export"
        unique_id: enphase_grid_export
        state_class: measurement
        unit_of_measurement: W
        device_class: power
        state: >
            {{ [ 0, state_attr('sensor.enphase', 'consumption')[1]['wNow'] * -1 ] | max }}

[catsmanac]

Are you saying that the integration is not correctly reading the energy for net-consumption or that envoy is not providing it? Can you share a json output of the production.json? I have no meters so can't compare to code. I'm using Rieman for my individual panels and results are very close to what enphase website and envoy totals report so that works really fine indeed.