Missing all "net" entities and consumption amps entities, but CTs are installed and enabled

I've been having this problem ever since installing this integration, but finally decided to post about it since it seems like recent updates didn't happen to fix it.

Basically everything works GREAT, I love this integration! :) But....it seems like I'm missing the following entities:

sensor.Envoy_<sn>_current_net_power_consumption
sensor.Envoy_<sn>_lifetime_net_energy_production
sensor.Envoy_<sn>_lifetime_net_energy_consumption
sensor.Envoy_<sn>_current_net_power_consumption_l<n>
sensor.Envoy_<sn>_lifetime_net_energy_production_l<n>
sensor.Envoy_<sn>_lifetime_net_energy_consumption_l<n>
sensor.Envoy_<sn>_consumption_Current
sensor.Envoy_<sn>_consumption_Current_l<n>

There are no hidden or disabled entities in my setup, so I'm positive I haven't overlooked these.

I do have working entities for instantaneous/7-day/lifetime consumption and it's accurate and matches what I see in the Enphase app. I also downloaded the diagnostics and tried to look through them for disabled meters as discovered in a few other issues here. But as far as I can tell it shows mine as enabled.

        "Endpoint-meters": "[\n    {\n        \"eid\": 704643328,\n        \"state\": \"enabled\",\n        \"measurementType\": \"production\",\n        \"phaseMode\": \"split\",\n        \"phaseCount\": 2,\n        \"meteringStatus\": \"normal\",\n        \"statusFlags\": []\n    },\n    {\n        \"eid\": 704643584,\n        \"state\": \"enabled\",\n        \"measurementType\": \"total-consumption\",\n        \"phaseMode\": \"split\",\n        \"phaseCount\": 2,\n        \"meteringStatus\": \"normal\",\n        \"statusFlags\": []\n    }\n]",
        "Endpoint-meters-readings": "[\n    {\n        \"eid\": 704643328,\n        \"timestamp\": 1708535673,\n        \"actEnergyDlvd\": 12209702.219,\n        \"actEnergyRcvd\": 1697.711,\n        \"apparentEnergy\": 17829362.369,\n        \"reactEnergyLagg\": 3233146.911,\n        \"reactEnergyLead\": 647840.583,\n        \"instantaneousDemand\": 5335.441,\n        \"activePower\": 5335.441,\n        \"apparentPower\": 5908.024,\n        \"reactivePower\": 15.662,\n        \"pwrFactor\": 0.902,\n        \"voltage\": 246.199,\n        \"current\": 47.996,\n        \"freq\": 60.000,\n        \"channels\": [\n            {\n                \"eid\": 1778385169,\n                \"timestamp\": 1708535673,\n                \"actEnergyDlvd\": 6104222.965,\n                \"actEnergyRcvd\": 863.515,\n                \"apparentEnergy\": 8911189.409,\n                \"reactEnergyLagg\": 1617316.320,\n                \"reactEnergyLead\": 321379.309,\n                \"instantaneousDemand\": 2669.357,\n                \"activePower\": 2669.357,\n                \"apparentPower\": 2953.904,\n                \"reactivePower\": 9.324,\n                \"pwrFactor\": 0.902,\n                \"voltage\": 123.127,\n                \"current\": 23.993,\n                \"freq\": 60.000\n            },\n            {\n                \"eid\": 1778385170,\n                \"timestamp\": 1708535673,\n                \"actEnergyDlvd\": 6105479.254,\n                \"actEnergyRcvd\": 834.196,\n                \"apparentEnergy\": 8918172.959,\n                \"reactEnergyLagg\": 1615830.591,\n                \"reactEnergyLead\": 326461.274,\n                \"instantaneousDemand\": 2666.084,\n                \"activePower\": 2666.084,\n                \"apparentPower\": 2954.120,\n                \"reactivePower\": 6.339,\n                \"pwrFactor\": 0.902,\n                \"voltage\": 123.072,\n                \"current\": 24.004,\n                \"freq\": 60.000\n            },\n            {\n                \"eid\": 1778385171,\n                \"timestamp\": 1708535673,\n                \"actEnergyDlvd\": 0.000,\n                \"actEnergyRcvd\": 0.000,\n                \"apparentEnergy\": 0.000,\n                \"reactEnergyLagg\": 0.000,\n                \"reactEnergyLead\": 0.000,\n                \"instantaneousDemand\": 0.000,\n                \"activePower\": 0.000,\n                \"apparentPower\": 0.000,\n                \"reactivePower\": 0.000,\n                \"pwrFactor\": 0.000,\n                \"voltage\": 0.000,\n                \"current\": 0.000,\n                \"freq\": 60.000\n            }\n        ]\n    },\n    {\n        \"eid\": 704643584,\n        \"timestamp\": 1708535673,\n        \"actEnergyDlvd\": 9779697.983,\n        \"actEnergyRcvd\": 111.852,\n        \"apparentEnergy\": 11728905.986,\n        \"reactEnergyLagg\": 1066004.932,\n        \"reactEnergyLead\": 2551489.384,\n        \"instantaneousDemand\": 2201.690,\n        \"activePower\": 2201.690,\n        \"apparentPower\": 3459.050,\n        \"reactivePower\": -713.075,\n        \"pwrFactor\": 0.636,\n        \"voltage\": 246.224,\n        \"current\": 28.068,\n        \"freq\": 60.000,\n        \"channels\": [\n            {\n                \"eid\": 1778385425,\n                \"timestamp\": 1708535673,\n                \"actEnergyDlvd\": 4771502.821,\n                \"actEnergyRcvd\": 47.263,\n                \"apparentEnergy\": 5642426.032,\n                \"reactEnergyLagg\": 669427.910,\n                \"reactEnergyLead\": 891946.099,\n                \"instantaneousDemand\": 1052.004,\n                \"activePower\": 1052.004,\n                \"apparentPower\": 1692.026,\n                \"reactivePower\": -297.957,\n                \"pwrFactor\": 0.621,\n                \"voltage\": 123.152,\n                \"current\": 13.733,\n                \"freq\": 60.000\n            },\n            {\n                \"eid\": 1778385426,\n                \"timestamp\": 1708535673,\n                \"actEnergyDlvd\": 5008195.161,\n                \"actEnergyRcvd\": 64.589,\n                \"apparentEnergy\": 6086479.954,\n                \"reactEnergyLagg\": 396577.022,\n                \"reactEnergyLead\": 1659543.285,\n                \"instantaneousDemand\": 1149.686,\n                \"activePower\": 1149.686,\n                \"apparentPower\": 1767.024,\n                \"reactivePower\": -415.117,\n                \"pwrFactor\": 0.651,\n                \"voltage\": 123.073,\n                \"current\": 14.334,\n                \"freq\": 60.000\n            },\n            {\n                \"eid\": 1778385427,\n                \"timestamp\": 1708535673,\n                \"actEnergyDlvd\": 0.000,\n                \"actEnergyRcvd\": 0.000,\n                \"apparentEnergy\": 1860.722,\n                \"reactEnergyLagg\": 431.264,\n                \"reactEnergyLead\": 0.537,\n                \"instantaneousDemand\": 0.000,\n                \"activePower\": 0.000,\n                \"apparentPower\": 0.354,\n                \"reactivePower\": -0.000,\n                \"pwrFactor\": 0.000,\n                \"voltage\": 3.221,\n                \"current\": 0.111,\n                \"freq\": 60.000\n            }\n        ]\n    }\n]",
        "Endpoint-meters-reports": "[\n    {\n        \"createdAt\": 1708535674,\n        \"reportType\": \"production\",\n        \"cumulative\": {\n            \"currW\": 5335.441,\n            \"actPower\": 5335.441,\n            \"apprntPwr\": 5908.024,\n            \"reactPwr\": 15.662,\n            \"whDlvdCum\": 12209702.219,\n            \"whRcvdCum\": 1697.711,\n            \"varhLagCum\": 3233146.911,\n            \"varhLeadCum\": 647840.583,\n            \"vahCum\": 17829362.369,\n            \"rmsVoltage\": 246.199,\n            \"rmsCurrent\": 47.996,\n            \"pwrFactor\": 0.90,\n            \"freqHz\": 60.00\n        },\n        \"lines\": [\n            {\n                \"currW\": 2669.357,\n                \"actPower\": 2669.357,\n                \"apprntPwr\": 2953.904,\n                \"reactPwr\": 9.324,\n                \"whDlvdCum\": 6104222.965,\n                \"whRcvdCum\": 863.515,\n                \"varhLagCum\": 1617316.320,\n                \"varhLeadCum\": 321379.309,\n                \"vahCum\": 8911189.409,\n                \"rmsVoltage\": 123.127,\n                \"rmsCurrent\": 23.993,\n                \"pwrFactor\": 0.90,\n                \"freqHz\": 60.00\n            },\n            {\n                \"currW\": 2666.084,\n                \"actPower\": 2666.084,\n                \"apprntPwr\": 2954.120,\n                \"reactPwr\": 6.339,\n                \"whDlvdCum\": 6105479.254,\n                \"whRcvdCum\": 834.196,\n                \"varhLagCum\": 1615830.591,\n                \"varhLeadCum\": 326461.274,\n                \"vahCum\": 8918172.959,\n                \"rmsVoltage\": 123.072,\n                \"rmsCurrent\": 24.004,\n                \"pwrFactor\": 0.90,\n                \"freqHz\": 60.00\n            }\n        ]\n    },\n    {\n        \"createdAt\": 1708535674,\n        \"reportType\": \"net-consumption\",\n        \"cumulative\": {\n            \"currW\": -3133.751,\n            \"actPower\": -3133.751,\n            \"apprntPwr\": 3455.451,\n            \"reactPwr\": 0.000,\n            \"whDlvdCum\": -1816219.976,\n            \"whRcvdCum\": 0.000,\n            \"varhLagCum\": -2167141.979,\n            \"varhLeadCum\": 1903648.801,\n            \"vahCum\": -6100456.383,\n            \"rmsVoltage\": 246.224,\n            \"rmsCurrent\": 19.929,\n            \"pwrFactor\": -0.90,\n            \"freqHz\": 60.00\n        },\n        \"lines\": [\n            {\n                \"currW\": -1617.353,\n                \"actPower\": -1617.353,\n                \"apprntPwr\": 1691.295,\n                \"reactPwr\": 0.000,\n                \"whDlvdCum\": -1026041.253,\n                \"whRcvdCum\": 0.000,\n                \"varhLagCum\": -947888.410,\n                \"varhLeadCum\": 570566.790,\n                \"vahCum\": -3268763.377,\n                \"rmsVoltage\": 123.152,\n                \"rmsCurrent\": 10.259,\n                \"pwrFactor\": 0.62,\n                \"freqHz\": 60.00\n            },\n            {\n                \"currW\": -1516.398,\n                \"actPower\": -1516.398,\n                \"apprntPwr\": 1764.156,\n                \"reactPwr\": 0.000,\n                \"whDlvdCum\": -790178.723,\n                \"whRcvdCum\": 0.000,\n                \"varhLagCum\": -1219253.569,\n                \"varhLeadCum\": 1333082.011,\n                \"vahCum\": -2831693.005,\n                \"rmsVoltage\": 123.073,\n                \"rmsCurrent\": 9.669,\n                \"pwrFactor\": 0.65,\n                \"freqHz\": 60.00\n            }\n        ]\n    },\n    {\n        \"createdAt\": 1708535674,\n        \"reportType\": \"total-consumption\",\n        \"cumulative\": {\n            \"currW\": 2201.690,\n            \"actPower\": 2201.690,\n            \"apprntPwr\": 3455.451,\n            \"reactPwr\": -713.075,\n            \"whDlvdCum\": 10393482.243,\n            \"whRcvdCum\": 0.000,\n            \"varhLagCum\": 1066004.932,\n            \"varhLeadCum\": 2551489.384,\n            \"vahCum\": 11728905.986,\n            \"rmsVoltage\": 246.224,\n            \"rmsCurrent\": 28.068,\n            \"pwrFactor\": 0.64,\n            \"freqHz\": 60.00\n        },\n        \"lines\": [\n            {\n                \"currW\": 1052.004,\n                \"actPower\": 1052.004,\n                \"apprntPwr\": 1691.295,\n                \"reactPwr\": -297.957,\n                \"whDlvdCum\": 5078181.712,\n                \"whRcvdCum\": 0.000,\n                \"varhLagCum\": 669427.910,\n                \"varhLeadCum\": 891946.099,\n                \"vahCum\": 5642426.032,\n                \"rmsVoltage\": 123.152,\n                \"rmsCurrent\": 13.733,\n                \"pwrFactor\": 0.62,\n                \"freqHz\": 60.00\n            },\n            {\n                \"currW\": 1149.686,\n                \"actPower\": 1149.686,\n                \"apprntPwr\": 1764.156,\n                \"reactPwr\": -415.117,\n                \"whDlvdCum\": 5315300.531,\n                \"whRcvdCum\": 0.000,\n                \"varhLagCum\": 396577.022,\n                \"varhLeadCum\": 1659543.285,\n                \"vahCum\": 6086479.954,\n                \"rmsVoltage\": 123.073,\n                \"rmsCurrent\": 14.334,\n                \"pwrFactor\": 0.65,\n                \"freqHz\": 60.00\n            }\n        ]\n    }\n]",

Further up in the diagnostics though, I see:

    "data": {
      "production": 5335,
      "daily_production": 5082,
      "seven_days_production": 12204983,
      "lifetime_production": 12209702,
      "lifetime_net_production": "CTs consumption data not available for your Envoy device.",
      "consumption": 2201,
      "net_consumption": "CTs consumption data not available for your Envoy device.",
      "daily_consumption": 36623,
      "seven_days_consumption": 10356979,
      "lifetime_consumption": 10393482,
      "lifetime_net_consumption": "CTs consumption data not available for your Envoy device.",

I've attached my diagnostic file, and would love to hear your opinions on the contents. It would be super awesome to get these entities up and running since I have the data in the enphase app, but not in HA where I can make use of it in fun ways. :(

Thanks!!!

config_entry-enphase_envoy-c1d2ce5abad566e96ccdaef6628904ee.json

Also here are a couple of screenshots from the Enphase Enlighten system showing my meters are both set up and working. Although now that I look at these I'm seeing that the consumption meter is installed as "load only" which your docs explain will omit the "net" entities.....dang! Is this really a limitation on supporting these, or just a preference? There was no way for us to physically instal the CTs in load with solar mode, since my combined meter/breaker panel doesn't have a space to place the CTs in such a location.

Hi @jimmyz80, looking at your data in the "envoy_info": section I see that the consumption CT is configured as total-consumption. (measurementType) The integration expects a net-consumption type to report grid import and export. (See also 180)

"Endpoint-meters": [{
        "eid": 704643328,
        "state": "enabled",
        "measurementType": "production",
        "phaseMode": "split",
        "phaseCount": 2,
        "meteringStatus": "normal",
        "statusFlags": []
    }, {
        "eid": 704643584,
        "state": "enabled",
        "measurementType": "total-consumption",
        "phaseMode": "split",
        "phaseCount": 2,
        "meteringStatus": "normal",
        "statusFlags": []
    }
],

That's why the "CTs consumption data not available for your Envoy device." is showing. So the first question is if the consumption CT is indeed a total-consumption one or a net-consumption one.

Looking at afbeelding it is indeed a total-consumption one as indicated by load-only.

Is this really a limitation on supporting these, or just a preference?

Unless we can find a recipe how to calculate from the 2 other CT's or the other data, its a limitation.

This is an excerpt of the CT meters readings

"Endpoint-meters-readings": [{
        "actEnergyDlvd": 12209702.219,
        "actEnergyRcvd": 1697.711,
        "activePower": 5335.441,
    }, {
        "actEnergyDlvd": 9779697.983,
        "actEnergyRcvd": 111.852,
        "activePower": 2201.690,
    }
]

First one is the production CT, second the total-consumption. For production CT actEnergyDlvd is from solar to switchboard, actEnergyRcvd the other direction. The 12.2MWH seems to match the Enphase screenshot.

For total-consumption CT actEnergyDlvd is to the house, 9.7MWh. But we just don't know how much of the solar went into the house or to the grid. And the number differs quite a bit from the 11MWh in your screenshots.

Back to clarify a few things and add a few thoughts... :)

This is indeed a load-only consumption CT. It's installed around all of the load wires exiting the panel to circuits in the house (other than the solar backfeed circuit).

For total-consumption CT actEnergyDlvd is to the house, 9.7MWh. But we just don't know how much of the solar went into the house or to the grid.

I think the "we" here might be the key. The Envoy certainly does, since at any given time it just needs to subtract the solar CT from the consumption CT. Anything negative is the current net kW being imported and anything positive is the net kW being exported. I can see it doing this in realtime on my status screen.

I can also see the same thing in HA just by calculating this difference between envoy__current_powerconsumption and envoy_current_power_production.

So over the lifetime of the system a simple difference between production and consumption will give you the lifetime net import or export. Enphase agrees with this on my lifetime data screen, where I can see 12.2MWh produced minus 9.7MWh consumed = 2.4MWh "net exported".

The "imported" and "exported" numbers on that screenshot through I believe are calculated internal to the Envoy using its database of historical datapoints for either importing or exporting (every 15min?, 1min?, whatever granularity they use for the internal DB). For clarity I would call these total export and total import because again when you calculate their difference you end up at my net export value. 8.6MWh Total Export - 6.2MWh Total Import = 2.4MWh Net Export. In any case, I don't believe these figures would be available externally in either of the CT setting modes.

It seems to me that the integration could fill in sensor.Envoy__current_net_powerconsumption just by computing the difference between envoy_current_powerconsumption and envoy_current_power_production. That's what the Enphase software has to be doing when the CT is in load-only mode (not sure why they only use it internally and don't expose it on the API though...). The Envoy just doesn't have to do this math when the CT is in load-plus-solar mode since then it can read that value directly by virtual of "mechanical calculation" of electrons flowing in both directions through a single CT.

Similarly for:

sensor.Envoy_<sn>_lifetime_net_energy_production
sensor.Envoy_<sn>_lifetime_net_energy_consumption

It seems like you would only ever have data in one of these but not both if they're truly "net", correct? I obviously can't see them on my setup so I can't compare to the Enphase values. But by seeing the word "net" in them I would expect to see sensor.Envoy__lifetime_net_energyconsumption be zero and sensor.Envoy_lifetime_net_energyproduction be 2.4MWh. If instead they ended up being populated by 8.6MWh and 6.2MWh that I earlier referred to as my TOTAL export and import, that seems like an error to me. "Net" would mean the difference calculation has been completed and you ended up with a single positive or negative number which would dictate whether it's written into sensor.Envoy_lifetime_net_energyproduction or sensor.Envoy_lifetime_net_energy_consumption and the other would be zero. Otherwise they're just totals of any detected periods of exports or imports, NOT yet netted out.

Sorry about this being so long winded, and hopefully I was able to articulate things in a somewhat clear fashion. :)

Not entirely relevant to my original topic, but I'm attaching a file that seems to confirm my above hunch that internally the Envoy was using 15min snapshots to create their total export and total import values for any window of time. This was exported from the Enphase Enlighten app. Notice that in every row there's only ever a value in the exported OR imported column but never BOTH. So they just take the production and consumption totals for a 15min window, compute the difference, and then determine if the result goes into the import or export column by checking if the result is positive or negative.

So this total import/export data is clearly something the integration can't provide unless the Enphase API does the math and exposes these totaled up for any given window of time. But I still feel the lifetime NET import and export data, and current net consumption could be provided as I described in my previous post. 4395521_monthly_net_energy_report.csv

I think the "we" here might be the key

Yes you are right. The envoy pushes a blob of data every so often for 15 min time periods and it buffers it if there's network issues. We, mere mortals, can't access that blob.

I can also see the same thing in HA just by calculating this difference between envoycurrent_power_consumption and envoycurrent_power_production.

We're talking power now, and yes, the diff between sensor.envoy_1234567890121_current_power_production and sensor.envoy_1234567890121_current_power_consumption is sensor.envoy_123456789012_current_net_power_consumption.

You can make this calc in HA using a template
Looking at the code it's not overly complicated to do this, but you would need to supply datasets I can use for this case and regression testing
What is not 100% clear is if these values are snapshots or represent the average of a time interval.
- The individual inverter data seems to be a Wh average for a 5 min interval calculated every 5 min
- The current_power_production and current_net_power_consumption come from the production report and may be similar, but I'm not sure. We need to validate this.
- Using CT numbers may be just snapshot, so collecting these ever 1 minute may be inaccurate.

Back to Energy, That is internally in the Envoy integrated from the powers and calculated. As for 'we ...', we have no reference. If solar produced 12 and the house used 9 then any import/export value pairs that account for 3 net export up to max import of 9 would meet the numbers.

house	solar	import	export
9	12	0	3
9	12	1	4
9	12	2	5

Haven't delved through all other reports if we can find a reference. That's take some time.

Not sure if I addressed all your items. In summary your are right net power can be calculated and net energy can't unless we find some reference in existing reports. (the endpoint... lines in the diagnostic lines, or for multiple sets run with debug enabled over HA restart and couple of minutes)

briancmpbll / home_assistant_custom_envoy

Missing all "net" entities and consumption amps entities, but CTs are installed and enabled #184