Legacy Envoy stopped working in versions 0.0.12 & 0.0.13

[madbrain76]

Using an Envoy LCD (Envoy-R) I had to revert back to 0.0.11 to get it to work again. There is a regression in 0.0.12 that broke the legacy Envoy.

Note that also have an IQ Envoy in addition, but I don't think it matters. The IQ Envoy still works in versions 0.0.12/0.0.13 .

[DMedina559]

@catsmanac still no luck

Do I switch to one of your integrations for further testing?

[catsmanac]

@DMedina559 yes please switch your HACS to https://github.com/catsmanac/home_assistant_custom_envoy

I just updated the code for the next onion peel. Turns out that the logic to detect if grid-status is available was crashing as ENVOY R with your firmware returns error status and no data. I updated the logic to only use grid-status if home page was retrieved with success status 200.

Same process as before, add the debug to configuration.yaml and try again after loading the integration from my dev repository.

[DMedina559]

@catsmanac great news that worked

[catsmanac]

Ah excellent, quite a relief. Can you get me the log file again after it ran for a little bit so I can verify it indeed does all we expect it to do.

[DMedina559]

@catsmanac

[catsmanac]

I assume you are getting all the data you were looking for? From the log file it does collect but in a pattern that suggests it's trying to find out what type Envoy it is each time rather then just getting the data.

I was expecting to find something in the log like

....Using Model: P (HTTP, Metering enabled: False, Get Inverters: False)

but in the one you send it's not there. If the debug log is still running can you find such a line.

[DMedina559]

2023-07-26 12:16:10.300 DEBUG (MainThread) [custom_components.enphase_envoy.envoy_reader] Using Model: P (HTTP, Metering enabled: False, Get Inverters: False)

It looks like I'm getting the same 4 sensors the ha core integration gives

[catsmanac]

Thanks, the log looks fine. The first one just missed that model report and got me worried, but the last one is just excellent :-).

[DMedina559]

Awesome. Thank you so much for you work

[catsmanac]

I'll disable some added warnings and prepare for final inclusion into main. I might ask for a final last load then.

Thanks so much for helping on this. Work is on the way to promote this integration as the new HA Core version and your testing with the older model is helping the verification for the older models a great deal. These changes will be included there too and finally become the new core. If you are willing to join that testing in the future we would appreciate that as well.

[madbrain76]

I'm happy to continue testing in the future with all my Envoys.

[DMedina559]

@catsmanac Yeah for sure, whatever you need just let me know.

[catsmanac]

Thanks @madbrain76 and @DMedina559, I'll reach out when testing can be done. Process will be similar to this one, add another custom integration to HACS and download.

[catsmanac]

Debug cleanup is done and new version is available for final test in HACS repo https://github.com/catsmanac/home_assistant_custom_envoy before I create an update for the main.

[DMedina559]

Working perfectly, here's an updated log file.

[catsmanac]

Excellent thanks @DMedina559

@posixx would you mind testing it with your 3 phase before I create the PR?

[posixx]

I'm on vacation right now so don't want to fiddle with a running system remotely. Will be back around the 13th of august..

[catsmanac]

I'm on vacation right now so don't want to fiddle with a running system remotely. Will be back around the 13th of august..

Understand, enjoy your vacation.

[Steve2017]

@catsmanac I have just installed your latest version (updated from the original @briancmpbll version) My Envoy S Metered is using this: Software Version D7.6.175 (f79c8d)

So far it looks okay. I might have missed this, but I notice it creates new sensors with a new addition "_l1". For example -xxxxxxx_current_power_consumption_l1. It gives the same data as the one without "_l1". Is that for three phase? (I have single phase)

After sundown when the solar stops productions will be the test for me. With Brian's version I had a number of downstream template sensors become unavailable after production became zero. For example, I have a sensor that tells me how much solar has been self-consumed and the import cost avoided. That has been unavailable when the source sensor resets: It comes back when solar production resumes.

That might be an issue with my YAML, not Brian's integration, but we'll see. It used to be fine with the original BFU (before-firmware-update) integration.

[catsmanac]

Thanks for testing @Steve2017. as for

I might have missed this, but I notice it creates new sensors with a new addition "_l1".

Was that present when you ran V0.0.12 or v0.0.13 or did it just popup with this test version? Take a look at it's history to determine when it showed. Since you have a metered envoy I'm guessing it's detecting the single phase and adding it just as it would if it was 3 phase. @posixx can probably tell but he's enjoying a vacation right now.

For your solar use sensor

I have a sensor that tells me how much solar has been self-consumed and the import cost avoided

I would think a value of 0 is still a valid number, is there some logic protecting aginst value gowing down maybe?

You can use the energy dashboard in HA for this too. That's probably a different topic, let's first validate you are getting the right numbers.

[madbrain76]

I also have L1/L2 sensors on my two IQ Envoys even though I don't have 3-phase service. Not sure when those appeared exactly.

[Steve2017]

Was that present when you ran V0.0.12 or v0.0.13

Hi @catsmanac . I did not update to v 0.0.12 or v 0.0.13 because v 0.0.11 appeared to be working and I had other demands on my time. I went straight from that to your updated version. So - I can't answer the question about the appearance of the additional "_l1" sensors. I'll just disable them. Unless they interfere, it shouldn't be an issue, but understand that the issue does need further examination.

On the sensors becoming unavailable, I think the issue is with my YAML/config file and sensor formats, not your integration.

After 24 hours, I still have the same issues with the downstream sensors becoming unavailable after solar production drops to zero or the daily reset occurs. I need to go through my template sensors and see where the initial issue is. I suspect it might be where I have created a template sensor from an integration sensor which uses data from a utility meter, like this:

  - name: Solar Self-Consumed
    state_class: measurement
    icon: mdi:lightning-bolt
    unit_of_measurement: "kWh"
    device_class: power
    state: >
      {{ ((states('sensor.solar_daily') |float - 
            states('sensor.solar_export_daily') |float)) | round(3) }}

Since HA updates in April or May I have had some issues with utility meters, so I will play around with those. I might have missed an updated format change.

[catsmanac]

Had a quick look at the 3phase code added by @posixx and it will auto-detect if the envoy is reporting multiple phase data from the meters and created the entities or not. So very neatly. First intent is obviously to use with 3 phase grid home installations, but technically it is multi-phase. It will work as well when using single phase service but with panels and meters behind different breakers all connected to same grid phase. If only 2 phases are measured there will only be L1/L2 and if only 1 phase L1. And I expect it's a configuration of the Envoy how many meters are configured to report, not sure, I don't have the metered version.

You can see in the debug data what the envoy is reporting. Look for a line containing:

Fetched from https://192.168.01.10/production.json?details=1: <Response [200 OK]>:

There will be a long line behind it that contains the details in json. Look for "lines": in there and see how often these occur. There will be such a lines section for production and consumption. On below example I did a bit of formatting so it's better readable and you see 2 lines for both production and consumption and for net-consumpion If there's multiple lines measured, the entity without _L1/_L2 will be the sum of the others. In case of a single lines section both will be the same.

{
    "production": [
        {
            "type": "inverters",
            "activeCount": 19,
            "readingTime": 1690295737,
            "wNow": 525,
            "whLifetime": 379457
        },
        {
            "type": "eim",
            "activeCount": 1,
            "measurementType": "production",
            "readingTime": 1690295758,
            "wNow": 591.738,
            "whLifetime": 2068955.25,
            "varhLeadLifetime": 4628.85,
            "varhLagLifetime": 291450.106,
            "vahLifetime": 2320528.639,
            "rmsCurrent": 4.925,
            "rmsVoltage": 242.17,
            "reactPwr": -5.245,
            "apprntPwr": 596.359,
            "pwrFactor": 1.0,
            "whToday": 297.25,
            "whLastSevenDays": 222074.25,
            "vahToday": 3033.639,
            "varhLeadToday": 3.85,
            "varhLagToday": 2737.106,
            "lines": [
                {
                    "wNow": 296.316,
                    "whLifetime": 1028298.771,
                    "varhLeadLifetime": 2628.982,
                    "varhLagLifetime": 144590.875,
                    "vahLifetime": 1153512.67,
                    "rmsCurrent": 2.461,
                    "rmsVoltage": 121.25,
                    "reactPwr": -2.187,
                    "apprntPwr": 298.363,
                    "pwrFactor": 1.0,
                    "whToday": 148.771,
                    "whLastSevenDays": 110699.771,
                    "vahToday": 1511.67,
                    "varhLeadToday": 1.982,
                    "varhLagToday": 1363.875
                },
                {
                    "wNow": 295.423,
                    "whLifetime": 1040656.479,
                    "varhLeadLifetime": 1999.867,
                    "varhLagLifetime": 146859.231,
                    "vahLifetime": 1167015.969,
                    "rmsCurrent": 2.464,
                    "rmsVoltage": 120.92,
                    "reactPwr": -3.058,
                    "apprntPwr": 297.996,
                    "pwrFactor": 1.0,
                    "whToday": 148.479,
                    "whLastSevenDays": 111374.479,
                    "vahToday": 1521.969,
                    "varhLeadToday": 1.867,
                    "varhLagToday": 1373.231
                }
            ]
        }
    ],
    "consumption": [
        {
            "type": "eim",
            "activeCount": 0,
            "measurementType": "total-consumption",
            "readingTime": 1690295758,
            "wNow": 362.736,
            "whLifetime": 769671.269,
            "varhLeadLifetime": 68953.161,
            "varhLagLifetime": 353975.054,
            "vahLifetime": 374413.471,
            "rmsCurrent": 2.453,
            "rmsVoltage": 242.198,
            "reactPwr": 107.258,
            "apprntPwr": 594.19,
            "pwrFactor": 0.61,
            "whToday": 769671.269,
            "whLastSevenDays": 769671.269,
            "vahToday": 374413.471,
            "varhLeadToday": 68953.161,
            "varhLagToday": 353975.054,
            "lines": [
                {
                    "wNow": 198.0,
                    "whLifetime": 444267.864,
                    "varhLeadLifetime": 43963.806,
                    "varhLagLifetime": 181875.924,
                    "vahLifetime": 267978.053,
                    "rmsCurrent": 1.104,
                    "rmsVoltage": 121.258,
                    "reactPwr": 78.983,
                    "apprntPwr": 133.868,
                    "pwrFactor": 1.0,
                    "whToday": 444267.864,
                    "whLastSevenDays": 444267.864,
                    "vahToday": 267978.053,
                    "varhLeadToday": 43963.806,
                    "varhLagToday": 181875.924
                },
                {
                    "wNow": 164.736,
                    "whLifetime": 325403.404,
                    "varhLeadLifetime": 24989.355,
                    "varhLagLifetime": 172099.13,
                    "vahLifetime": 106435.418,
                    "rmsCurrent": 1.349,
                    "rmsVoltage": 120.94,
                    "reactPwr": 28.275,
                    "apprntPwr": 163.187,
                    "pwrFactor": 1.0,
                    "whToday": 325403.404,
                    "whLastSevenDays": 325403.404,
                    "vahToday": 106435.418,
                    "varhLeadToday": 24989.355,
                    "varhLagToday": 172099.13
                }
            ]
        },
        {
            "type": "eim",
            "activeCount": 0,
            "measurementType": "net-consumption",
            "readingTime": 1690295758,
            "wNow": -229.002,
            "whLifetime": 209416.006,
            "varhLeadLifetime": 64324.311,
            "varhLagLifetime": 62524.948,
            "vahLifetime": 374413.471,
            "rmsCurrent": 2.471,
            "rmsVoltage": 242.226,
            "reactPwr": 102.013,
            "apprntPwr": 299.526,
            "pwrFactor": -0.78,
            "whToday": 0,
            "whLastSevenDays": 0,
            "vahToday": 0,
            "varhLeadToday": 0,
            "varhLagToday": 0,
            "lines": [
                {
                    "wNow": -98.316,
                    "whLifetime": 174103.066,
                    "varhLeadLifetime": 41334.824,
                    "varhLagLifetime": 37285.049,
                    "vahLifetime": 267978.053,
                    "rmsCurrent": 1.357,
                    "rmsVoltage": 121.267,
                    "reactPwr": 76.796,
                    "apprntPwr": 164.787,
                    "pwrFactor": -0.61,
                    "whToday": 0,
                    "whLastSevenDays": 0,
                    "vahToday": 0,
                    "varhLeadToday": 0,
                    "varhLagToday": 0
                },
                {
                    "wNow": -130.687,
                    "whLifetime": 35312.94,
                    "varhLeadLifetime": 22989.488,
                    "varhLagLifetime": 25239.899,
                    "vahLifetime": 106435.418,
                    "rmsCurrent": 1.114,
                    "rmsVoltage": 120.959,
                    "reactPwr": 25.217,
                    "apprntPwr": 134.739,
                    "pwrFactor": -1.0,
                    "whToday": 0,
                    "whLastSevenDays": 0,
                    "vahToday": 0,
                    "varhLeadToday": 0,
                    "varhLagToday": 0
                }
            ]
        }
    ],
    "storage": [
        {
            "type": "acb",
            "activeCount": 0,
            "readingTime": 0,
            "wNow": 0,
            "whNow": 0,
            "state": "idle"
        }
    ]
}

[catsmanac]

unit_of_measurement: "kWh" device_class: power

@Steve2017 looking at your setup I wonder if device class is the issue. HA Description for device class power states states unit_of_measurement for power should be W or kW and you specify kWh which is for device_class energy. Maybe that's causing different behavior?

[Steve2017]

@Steve2017 looking at your setup I wonder if device class is the issue. HA Description for device class power states states unit_of_measurement for power should be W or kW and you specify kWh which is for device_class energy. Maybe that's causing different behavior?

Yes - I was just looking at that and wondering why I had power and not energy. It could very well be the cause of my issues, especially when I have one dependent on another. I am also going through my Utility Meter format to make sure that is not throwing up errors.

[Steve2017]

@catsmanac My issue is either with the Riemann sum integral converting power data to energy - or my template sensor. For example, when the template sensor to record say "export power" drops to zero (on sundown) the Riemann sum sensor becomes "unavailable":

 platform: integration
 name: Solar Export Energy ## perpetual kWh
 source: sensor.solar_export_power ## from template sensor measuring Envoy Power
 unit_time: h
 method: left

This template sensor drops to zero at sunset but is still "available" There might be an issue with my calculation that makes it return a null.

   - name: Solar Export Power 
     state_class: measurement
     icon: mdi:transmission-tower
     unit_of_measurement: "kW"
     device_class: power
     state: >
       {{ [0, (states('sensor.envoy_12213500xxxx_current_power_production') | int - 
             states('sensor.envoy_12213500xxxx_current_power_consumption') | int)  /1000] | max }}

[Steve2017]

looking at your setup I wonder if device class is the issue.

@catsmanac That appears to have been the cause of my disappearing sensors. With the device class descriptions fixed, the problem has gone away, so it was nothing to do with the integration update.

[catsmanac]

I also have L1/L2 sensors on my two IQ Envoys even though I don't have 3-phase service. Not sure when those appeared exactly.

@madbrain76 is it just L1/L2 or is there also an L3? If just L1/L2, do you have 2 measurement CT installed on each Envoy or just 1?

[Steve2017]

looking at your setup I wonder if device class is the issue.

@catsmanac That appears to have been the cause of my disappearing sensors. With the device class descriptions fixed, the problem has gone away, so it was nothing to do with the integration update.

I spoke too soon. Back to the drawing board.

[catsmanac]

Too bad @Steve2017. I assume the basics are ok and the sensor.envoy_12213500xxxx_current_power_production and sensor.envoy_12213500xxxx_current_power_consumption are actually showing values all the time?

I guess using diff of today's energy production entities would yield the same without needing the integration? Our even lifetime energy and using an daily utility meter to get daily values.

[madbrain76]

@castmanac, no L3 . I believe I have just 1 CT on each Envoy, for production, not consumption.

[Steve2017]

the sensor.envoy_12213500xxxx_current_power_production and sensor.envoy_12213500xxxx_current_power_consumption are actually showing values all the time?

No - and that appears to be the issue. Those sensors become unavailable for about two minutes regularly around 23:00. The _sensor.solar_exportpower that is created from the difference between those two, remains unavailable until solar production resumes the next morning. I checked quite a few sensors created by the integration (eg microinvertors) and they all exhibit the same behaviour.

Interesting in parallel, I am running another solution that uses the REST/RESTful integration. The equivalent Envoy sensors also become unavailable, but it seems to happen later - at 01:01. They too remain unavailable for about 2-minutes. The sensor created from those production and consumption sensors also remains unavailable until solar production resumes.

I don't know whether this is just my Envoy, but perhaps someone else with an Envoy S Metered can check to see if they have a dropout. This is the power consumption sensor created by the integration, with the dropout.

The fact that the unavailability occurs at different times makes me wonder whether it is an Envoy log-in issue with tokens etc. (The REST version uses a token pasted in. It has 12 month validity)

[catsmanac]

Thanks for checking @madbrain76. Would you and @DMedina559 be willing to test Envoy_2_Core_custom_test to validate that version works with legacy models before we ask it to become new HA Core version? Just add it as custom repository https://github.com/catsmanac/Envoy_2_Core_custom_test to hacs and download it. To switch back to this one later re-download this one. Both are compatible for data.

[catsmanac]

The fact that the unavailability occurs at different times makes me wonder whether it is an Envoy log-in issue with tokens etc. (The REST version uses a token pasted in. It has 12 month validity)

I've seen more reports of Envoy brief outages when multiple clients are querying it. Envoy seems to run out of resources briefly. That's the reason that in Envoy_2_Core_custom_test I used a change in communication by @vincentwolsink used in his installer custom integration.

If you could try to test Envoy_2_Core_custom_test to see if that improves the situation I'd appreciate it.

Other thing you could try is add availability to your sensor like below. I think it won't calculate if that is not met.

availability: "{{ states(' sensor.envoy_12213500xxxx_current_power_production') | is_number }}"

[Steve2017]

If you could try to test Envoy_2_Core_custom_test to see if that improves the situation I'd appreciate it.

I am using that test integration and aside from those dropouts it is performing well. I'll try the availability suggestion in a test overnight tonight.

The other version of that might be?? availability: "{{ states('sensor.envoy_1221350xxxxx_current_power_production') | int(0) }}"

I'm floundering but my understanding is anything that can't be converted to an integer, (unavailable or unknown) will be given the default value of zero. That won't work for consumption, but maybe for production. I'll set up temporary sensors and see how it goes.

[catsmanac]

You may want to enable debug overnight to see what actually is happening and what status is coming back from the envoy.

[DMedina559]

@catsmanac just installed 0.2.5 and it's working perfectly fine with both my envoys

[catsmanac]

Thanks for testing @DMedina559, it's reassuring before core gets replaced.

[Steve2017]

You may want to enable debug overnight to see what actually is happening and what status is coming back from the envoy.

@catsmanac This is the relevant debug log. It starts at the end of the last successful GET at 22:59:28. That took just over one second. At 23:00:28 it starts another validation and GET, but at the 3rd and 2nd last line it fails with an ERROR and time-out after 30 seconds. This lines up with the history graph. The last line is the next attempt, which goes on to be successful.

> 2023-08-01 22:59:28.218 DEBUG (MainThread) [custom_components.enphase_envoy] Finished fetching envoy Envoy 1221xxxxx data in 1.092 seconds (success: True)
> 2023-08-01 23:00:28.127 DEBUG (MainThread) [custom_components.enphase_envoy.envoy_reader] Checking Token value: ******REPLACING TOKEN DATA*****
> 2023-08-01 23:00:28.128 DEBUG (MainThread) [custom_components.enphase_envoy.envoy_reader] Token is populated: ******REPLACING TOKEN DATA*****
> 2023-08-01 23:00:28.128 DEBUG (MainThread) [custom_components.enphase_envoy.envoy_reader] Token expires at: 2024-07-30 14:21:48
> 2023-08-01 23:00:28.128 DEBUG (MainThread) [custom_components.enphase_envoy.envoy_reader] HTTP GET Attempt #1: https://192.168.1.144/production.json?details=1: Header:{'Authorization': 'Bearer ******REPLACING TOKEN DATA*****'}
> 2023-08-01 23:00:58.130 ERROR (MainThread) [custom_components.enphase_envoy] Timeout fetching envoy Envoy 1221xxxxxx data
> 2023-08-01 23:00:58.135 DEBUG (MainThread) [custom_components.enphase_envoy] Finished fetching envoy Envoy 1221xxxxxxx data in 30.008 seconds (success: False)
> 2023-08-01 23:01:58.126 DEBUG (MainThread) [custom_components.enphase_envoy.envoy_reader] Checking Token value: 

I'm not sure if this is enough of the log, but my limited knowledge pushes me toward an Envoy issue rather than an integration issue.

What gets me is the template sensors that rely on the integration sensors. Why do they stay unavailable after the integration sensors come back on? I thought the availability command might fix that, but I guess I just don't understand its role. I'll take that elswhere and leave you to continue with the integration.

[catsmanac]

Would agree, or something on your network. As for the integrator, that is not my stronghold so indeed better ask at the forum.

[catsmanac]

@catsmanac just installed 0.2.5 and it's working perfectly fine with both my envoys

@DMedina559 0.2.6 is now available offering a download diagnostics button in the envoy device page and showing the firmware version for new envoy and not for old Envoy. And it should not break with old envoy which doesn't have firmware level. Would be great if you can test a last time. After that we'll start migrate to HA Core. Thanks.

[DMedina559]

error_log-2.txt

Envoy R isn't loading with this one

[catsmanac]

Thanks for trying, I'll have a look what is breaking it.

[catsmanac]

Sorry for the trouble @DMedina559 . I found the issue and updated the code. No new version generated yet. If you download again select the _Envoy_2_Core_As_CustomIntegration at the bottom of the list instead on 0.2.6 or 0.2.5. Thanks.

(It's there now)

[DMedina559]

Awesome that worked. Thank you

[catsmanac]

Excellent, thanks for testing. 0.2.7 published now.

[DMedina559]

@catsmanac I updated HA to 2023.08.0 and now I'm getting an error on the integration loading

[madbrain76]

I'm seeing the same problem with 2023.08.0 .

Logger: homeassistant.setup Source: setup.py:209 First occurred: 11:43:26 (1 occurrences) Last logged: 11:43:26

Setup failed for custom integration enphase_envoy: Requirements for enphase_envoy not found: ['pyjwt==2.7.0'].

[catsmanac]

Ah, I hadn't loaded 2023.8 yet. So next step.

I've opened an issue in the Envoy_2_Core_custom_test repo issues as this does not apply to @briancmpbll's integration. Let's track and discuss it there. Should have done that earlier, sorry @briancmpbll

[catsmanac]

@madbrain76 and @DMedina559 would you be willing to check if your legacy envoys can return xml data? I found out that HA is building a new HA Core one for Envoy which is completely different from @briancmpbll . It will require the envoy can return xml and not html.

In your browser can you open http://envoy-ip/info.xml and check if this returns xml and if serialnumber and/or lifetime production and/or firmware version and/or other data is in there? Or just post it here in a text file. And also try http://envoy-ip/info

If no xml it looks like you'll have to continue using @briancmpbll integration in the future as well. Thanks