PR to add a persistent token cache

[whisk3y]

This PR was based on another fork of this repo, I'm working on fixing the merge conflicts but this will allow for persistent token cache through restarts while there is an internet outage.

[whisk3y]

Fixed the merge conflicts

[enkrypt3d]

https://github.com/vk2him/Enphase-Envoy-mqtt-json/issues/5#issuecomment-1296373195 can we get this envoy_reader.py updated to include this curl to get a 1 year token? I tested it and it's working from the command line. Just need to have envoy_reader.py do it and cache it.

Edit: I was able to cache the token and then manually edit the token_cache.json with the 1 year token and that appears to be working.

[enkrypt3d]

Thanks! - it also looks like grid status is missing from this branch....

On Mon, Mar 27, 2023 at 9:56 PM Phil @.***> wrote:

I'll take a look tomorrow and see about adding the updates. I'll need to merge Brian's branch into mine, the token on my side is a 1 year token already I'll check out the curl and see what's different.

— Reply to this email directly, view it on GitHub https://github.com/briancmpbll/home_assistant_custom_envoy/pull/57#issuecomment-1486094987, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRWYXT4FNMQZDGJYKOLW6JAOXANCNFSM6AAAAAAWDGURAY . You are receiving this because you commented.Message ID: @.***>

[whisk3y]

@enkrypt3d Can you link to the section of code you think is missing? I merged with Brian's main branch and it's not showing anything missing.

[enkrypt3d]

https://github.com/briancmpbll/home_assistant_custom_envoy/blob/main/custom_components/enphase_envoy_custom/sensor.py

See the sensors batteries charged vs discharged. not sure how your merge didn't include these as it doesn't appear you changed sensor.py...... ?

On Tue, Mar 28, 2023 at 7:19 PM Whisk3y @.***> wrote:

@enkrypt3d https://github.com/enkrypt3d Can you link to the section of code you think is missing? I merged with Brian's main branch and it's not showing anything missing.

— Reply to this email directly, view it on GitHub https://github.com/briancmpbll/home_assistant_custom_envoy/pull/57#issuecomment-1487728131, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRRAJYXNBSD55LDX2LLW6NWWZANCNFSM6AAAAAAWDGURAY . You are receiving this because you were mentioned.Message ID: @.***>

[whisk3y]

Those sensors are there: https://github.com/briancmpbll/home_assistant_custom_envoy/pull/57/files#diff-3f26758a224441b1cf00904b607536130fe4645feb8878ac6010f4cbb9cfbb23L77

[enkrypt3d]

[image: image.png]

The sensors may be there actually but for some reason it's not pulling any data anymore :( This is from the energy dashboard settings.

On Tue, Mar 28, 2023 at 7:39 PM Whisk3y @.***> wrote:

Those sensors are there: https://github.com/briancmpbll/home_assistant_custom_envoy/pull/57/files#diff-3f26758a224441b1cf00904b607536130fe4645feb8878ac6010f4cbb9cfbb23L77

— Reply to this email directly, view it on GitHub https://github.com/briancmpbll/home_assistant_custom_envoy/pull/57#issuecomment-1487744028, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRUMO3ISGVMO7GCXBJ3W6NZBRANCNFSM6AAAAAAWDGURAY . You are receiving this because you were mentioned.Message ID: @.***>

[whisk3y]

@enkrypt3d Did you recently upgrade HA? I'm now getting this warning but didn't previously:

2023-03-28 20:23:54.899 WARNING (MainThread) [homeassistant.components.sensor] Entity sensor.envoy_122116060808_current_battery_capacity (<class 'custom_components.enphase_envoy.sensor.EnvoyEntity'>) is using native unit of measurement 'Wh' which is not a valid unit for the device class ('battery') it is using; expected one of ['%']; Please update your configuration if your entity is manually configured, otherwise report it to the custom integration author.

[whisk3y]

I see what you mean, they aren't in my fork, I just pushed them to main.

[enkrypt3d]

Thanks! - those entities are back. I'll give it some time to pull data from them into the energy dashboard....

On Tue, Mar 28, 2023 at 8:44 PM Whisk3y @.***> wrote:

I see what you mean, they aren't in my fork, I just pushed them to main.

— Reply to this email directly, view it on GitHub https://github.com/briancmpbll/home_assistant_custom_envoy/pull/57#issuecomment-1487787621, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRS64RTDPKDJYOCVBLTW6OAXLANCNFSM6AAAAAAWDGURAY . You are receiving this because you were mentioned.Message ID: @.***>

[enkrypt3d]

https://github.com/vk2him/Enphase-Envoy-mqtt-json

Wondering if we can fork this and get the near real time stats working again? It looks like he's almost got it working here but gave up..... just need to update the endpoint to envoy.localdomain/ivp/meters/readings and using the token auth and it should be near real time. This plus the token cache would help off grid scenarios...?

On Tue, Mar 28, 2023 at 9:22 PM Corey J @.***> wrote:

Thanks! - those entities are back. I'll give it some time to pull data from them into the energy dashboard....

On Tue, Mar 28, 2023 at 8:44 PM Whisk3y @.***> wrote:

I see what you mean, they aren't in my fork, I just pushed them to main.

— Reply to this email directly, view it on GitHub https://github.com/briancmpbll/home_assistant_custom_envoy/pull/57#issuecomment-1487787621, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRS64RTDPKDJYOCVBLTW6OAXLANCNFSM6AAAAAAWDGURAY . You are receiving this because you were mentioned.Message ID: @.***>

[catsmanac]

I was testing this PR by using whisk3y/home_assistant_envoy_d7_fw/tree/brianfork for a running @briancmpbll enphase envoy . After restart I got:

Error setting up entry Envoy 122302045041 for enphase_envoy

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 383, in async_setup
    result = await component.async_setup_entry(hass, self)
  File "/config/custom_components/enphase_envoy/__init__.py", line 37, in async_setup_entry
    token_cache_file=config[CONF_FILE_PATH],
KeyError: 'file_path'

Looks like CONF_FILE_PATH gets defined at setup of a new one. An existing one fails. Am I doing something wrong or would calls for use of async_migrate_entry in init.py?

[enkrypt3d]

Try deleting the integration and restart HA Then redo the HACS addon...

On Fri, Apr 14, 2023 at 3:09 PM Arie Catsman @.***> wrote:

I was testing this PR by using whisk3y/home_assistant_envoy_d7_fw/tree/brianfork https://github.com/whisk3y/home_assistant_envoy_d7_fw/tree/brianfork for a running @briancmpbll https://github.com/briancmpbll enphase envoy . After restart I got:

Error setting up entry Envoy 122302045041 for enphase_envoy

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/config_entries.py", line 383, in async_setup result = await component.async_setup_entry(hass, self) File "/config/custom_components/enphase_envoy/init.py", line 37, in async_setup_entry token_cache_file=config[CONF_FILE_PATH], KeyError: 'file_path'

Looks like CONF_FILE_PATH gets defined at setup of a new one. An existing one fails. Am I doing something wrong or would calls for use of async_migrate_entry https://developers.home-assistant.io/docs/config_entries_config_flow_handler/#config-entry-migration in init.py?

— Reply to this email directly, view it on GitHub https://github.com/briancmpbll/home_assistant_custom_envoy/pull/57#issuecomment-1509101310, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRRBE2QVQOOFN34F3IDXBGOHVANCNFSM6AAAAAAWDGURAY . You are receiving this because you were mentioned.Message ID: @.***>

[catsmanac]

Thanks, that worked. I now see the token_cache.json file as well.

[enkrypt3d]

I found these errors when the envoy stops responding and rebooting the envoy "fixes" it for a few weeks until it comes back....

2023-04-06 11:19:34.731 WARNING (MainThread) [homeassistant.components.sensor] Entity sensor.envoy_#########_current_battery_capacity (<class 'custom_components.enphase_envoy.sensor.TotalBatteryCapacityEntity'>) is using state class 'measurement' which is impossible considering device class ('energy') it is using; expected None or one of 'total_increasing', 'total'; Please update your configuration if your entity is manually configured, otherwise report it to the custom integration author.

and:

Logger: custom_components.enphase_envoy Source: helpers/update_coordinator.py:182 Integration: Enphase Envoy (DEV) First occurred: 10:20:58 AM (1 occurrences) Last logged: 10:20:58 AM

Error fetching envoy Envoy ########## data: Error communicating with API: peer closed connection without sending complete message body (incomplete chunked read)

This 2nd error appears when it cannot talk to the envoy anymore. Usually restarting the envoy resolves it for a while and then it comes back. I'm wondering if we can include this to get around this issue?

https://stackoverflow.com/questions/48105448/python-http-server-client-remote-end-closed-connection-without-response-error

self.send_header('Content-Type', 'blabla' )

self.end_headers()

[enkrypt3d]

Can anyone advise?

[catsmanac]

As for the first error/warning change this is probably an issue on it's own to report to fix this. Try updating const.py

    SensorEntityDescription(
        key="current_battery_capacity",
        name="Current Battery Capacity",
        native_unit_of_measurement=ENERGY_WATT_HOUR,
        state_class=SensorStateClass.MEASUREMENT,
        device_class=SensorDeviceClass.ENERGY
    ),

and change state_class to total

        state_class=SensorStateClass.TOTAL,

and see if that resolves it. I have no battery so I can't reproduce this one.

As for the second error and option you refer to, seems to me that the described option is on the server side, so that's the envoy web pages which are not in our control.

When it is in this state of peer closed can you still access the envoy webpages using browser or something like visual studio code with http add-in? I.e. try to determine is envoy web still works and HA is just for some reason (enabling debug on envoy would also show if HA is still reading the Envoy and what's coming back) or envoy web is all gone and requires the restart. Maybe HA needs to handle this case special and rebuild the session with the token and so.

[enkrypt3d]

I will have to find a way to manually probe the envoy next time this happens. The cloud UI for enphase manager continues to work just fine but HA can't connect to it anymore until I reboot the envoy.

On Thu, Apr 27, 2023 at 3:13 PM Arie Catsman @.***> wrote:

As for the first error/warning change this is probably an issue on it's own to report to fix this. Try updating const.py

SensorEntityDescription(
    key="current_battery_capacity",
    name="Current Battery Capacity",
    native_unit_of_measurement=ENERGY_WATT_HOUR,
    state_class=SensorStateClass.MEASUREMENT,
    device_class=SensorDeviceClass.ENERGY
),

and change state_class to total

    state_class=SensorStateClass.TOTAL,

and see if that resolves it. I have no battery so I can't reproduce this one.

As for the second error and option you refer to, seems to me that the described option is on the server side, so that's the envoy web pages which are not in our control.

When it is in this state of peer closed can you still access the envoy webpages using browser or something like visual studio code with http add-in? I.e. try to determine is envoy web still works and HA is just for some reason (enabling debug on envoy would also show if HA is still reading the Envoy and what's coming back) or envoy web is all gone and requires the restart. Maybe HA needs to handle this case special and rebuild the session with the token and so.

— Reply to this email directly, view it on GitHub https://github.com/briancmpbll/home_assistant_custom_envoy/pull/57#issuecomment-1526212515, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRSYXXJFF55IRNMVV3DXDLALZANCNFSM6AAAAAAWDGURAY . You are receiving this because you were mentioned.Message ID: @.***>

[catsmanac]

When in this state have you tried reloading the envoy integration using the reload option from the integration screen? That would build a fresh session. Or restart HA. And have debug enabled first it so you can see the details of what is was attempting before and after all these tests,

[catsmanac]

And for the first error, looking at the sensor description the device class for battery is probably ENERGY_STORAGE. In the description of total though it states: for example remaining battery capacity or CPU load; in such cases state class measurement should be used instead. This suggests to not change the unit to total. Const.py in core has

    SensorDeviceClass.ENERGY: {
        SensorStateClass.TOTAL,
        SensorStateClass.TOTAL_INCREASING,
    },
    SensorDeviceClass.ENERGY_STORAGE: {SensorStateClass.MEASUREMENT},

so it seems using ENERGY_STORAGE device class is the way to go.

[enkrypt3d]

Yes I have tried restarting HA and nothing seems to work beyond rebooting the envoy. I'll see if I can get more details next time it happens. thanks!

On Fri, Apr 28, 2023 at 5:09 AM Arie Catsman @.***> wrote:

And for the first error, looking at the sensor description https://developers.home-assistant.io/docs/core/entity/sensor/ the device class for battery is probably ENERGY_STORAGE. In the description of total though it states: for example remaining battery capacity or CPU load; in such cases state class measurement should be used instead. This suggests to not change the unit to total. Const.py in core has

SensorDeviceClass.ENERGY: {
    SensorStateClass.TOTAL,
    SensorStateClass.TOTAL_INCREASING,
},
SensorDeviceClass.ENERGY_STORAGE: {SensorStateClass.MEASUREMENT},

so it seems using ENERGY_STORAGE device class is the way to go.

— Reply to this email directly, view it on GitHub https://github.com/briancmpbll/home_assistant_custom_envoy/pull/57#issuecomment-1527240237, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRSC7J3PVVB4KAGXYXDXDOCOBANCNFSM6AAAAAAWDGURAY . You are receiving this because you were mentioned.Message ID: @.***>

[enkrypt3d]

Here is the latest crash:

Logger: homeassistant.config_entries Source: custom_components/enphase_envoy/envoy_reader.py:195 Integration: Enphase Envoy (DEV) First occurred: 11:29:32 AM (1 occurrences) Last logged: 11:29:32 AM

Error setting up entry Envoy 2######### for enphase_envoy Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/anyio/streams/tls.py", line 130, in _call_sslobject_method result = func(*args) File "/usr/local/lib/python3.10/ssl.py", line 917, in read v = self._sslobj.read(len) ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:2578)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/config_entries.py", line 387, in async_setup result = await component.async_setup_entry(hass, self) File "/config/custom_components/enphase_envoy/init.py", line 114, in async_setup_entry await coordinator.async_config_entry_first_refresh() File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 226, in async_config_entry_first_refresh await self._async_refresh( File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 258, in _async_refresh self.data = await self._async_update_data() File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 217, in _async_update_data return await self.update_method() File "/config/custom_components/enphase_envoy/init.py", line 49, in async_update_data await envoy_reader.getData() File "/config/custom_components/enphase_envoy/envoy_reader.py", line 384, in getData await self.detect_model() File "/config/custom_components/enphase_envoy/envoy_reader.py", line 419, in detect_model await self._update_from_pc_endpoint() File "/config/custom_components/enphase_envoy/envoy_reader.py", line 160, in _update_from_pc_endpoint await self._update_endpoint( File "/config/custom_components/enphase_envoy/envoy_reader.py", line 179, in _update_endpoint response = await self._async_fetch_with_retry( File "/config/custom_components/enphase_envoy/envoy_reader.py", line 195, in _async_fetch_with_retry resp = await client.get( File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1754, in get return await self.request( File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1530, in request return await self.send(request, auth=auth, follow_redirects=follow_redirects) File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1617, in send response = await self._send_handling_auth( File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1645, in _send_handling_auth response = await self._send_handling_redirects( File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects response = await self._send_single_request(request) File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1719, in _send_single_request response = await transport.handle_async_request(request) File "/usr/local/lib/python3.10/site-packages/httpx/_transports/default.py", line 353, in handle_async_request resp = await self._pool.handle_async_request(req) File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request raise exc File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request response = await connection.handle_async_request(request) File "/usr/local/lib/python3.10/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request return await self._connection.handle_async_request(request) File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 116, in handle_async_request raise exc File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 95, in handle_async_request ) = await self._receive_response_headers(**kwargs) File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 159, in _receive_response_headers event = await self._receive_event(timeout=timeout) File "/usr/local/lib/python3.10/site-packages/httpcore/_async/http11.py", line 195, in _receive_event data = await self._network_stream.read( File "/usr/local/lib/python3.10/site-packages/httpcore/backends/asyncio.py", line 34, in read return await self._stream.receive(max_bytes=max_bytes) File "/usr/local/lib/python3.10/site-packages/anyio/streams/tls.py", line 195, in receive data = await self._call_sslobject_method(self._ssl_object.read, max_bytes) File "/usr/local/lib/python3.10/site-packages/anyio/streams/tls.py", line 137, in _call_sslobject_method data = await self.transport_stream.receive() File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 1265, in receive await self._protocol.read_event.wait() File "/usr/local/lib/python3.10/asyncio/locks.py", line 214, in wait await fut asyncio.exceptions.CancelledError

Any ideas how to fix?

On Fri, Apr 28, 2023 at 11:32 AM Corey J @.***> wrote:

Yes I have tried restarting HA and nothing seems to work beyond rebooting the envoy. I'll see if I can get more details next time it happens. thanks!

On Fri, Apr 28, 2023 at 5:09 AM Arie Catsman @.***> wrote:

And for the first error, looking at the sensor description https://developers.home-assistant.io/docs/core/entity/sensor/ the device class for battery is probably ENERGY_STORAGE. In the description of total though it states: for example remaining battery capacity or CPU load; in such cases state class measurement should be used instead. This suggests to not change the unit to total. Const.py in core has

SensorDeviceClass.ENERGY: {
    SensorStateClass.TOTAL,
    SensorStateClass.TOTAL_INCREASING,
},
SensorDeviceClass.ENERGY_STORAGE: {SensorStateClass.MEASUREMENT},

so it seems using ENERGY_STORAGE device class is the way to go.

— Reply to this email directly, view it on GitHub https://github.com/briancmpbll/home_assistant_custom_envoy/pull/57#issuecomment-1527240237, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRSC7J3PVVB4KAGXYXDXDOCOBANCNFSM6AAAAAAWDGURAY . You are receiving this because you were mentioned.Message ID: @.***>

[catsmanac]

Can you confirm:

• you are using the code with the persistent token cache?
• this happened during continued running, not during some configuration activities?
• after the error all continues running or crashes?

Searches on the error

sl.SSLWantReadError: The operation did not complete (read) (_ssl.c:2578)

returns some suggestions that something in the ssl layers are not ready to read or write and need more read or write. The error (in persistent token cachec code) occurs in the try 3 times read loop

    async def _async_fetch_with_retry(self, url, **kwargs):
        """Retry 3 times to fetch the url if there is a transport error."""
        for attempt in range(3):
            _LOGGER.debug(
                "HTTP GET Attempt #%s: %s: Header:%s",
                attempt + 1,
                url,
                self._authorization_header,
            )
            try:
                async with self.async_client as client:
                    resp = await client.get(
                        url, headers=self._authorization_header, timeout=30, **kwargs
                    )
                    if resp.status_code == 401 and attempt < 2:
                        _LOGGER.debug(
                            "Received 401 from Envoy; refreshing token, attempt %s of 2",
                            attempt+1,
                            )
                        could_refresh_cookies = await self._refresh_token_cookies()
                        if not could_refresh_cookies:
                            await self._getEnphaseToken()
                        continue
                    _LOGGER.debug("Fetched from %s: %s: %s", url, resp, resp.text)
                    return resp
            except httpx.TransportError as e:
                _LOGGER.debug("TransportError: %s", e)
                if attempt == 2:
                    raise e

If any error other than transporterror occurs the loop just tries again and on 3th attempt raises the error. Looking around I've seen some examples that allow the sslwantreaderror to pass as in:

        except (ssl.SSLWantReadError, ssl.SSLWantWriteError):
            pass

Not sure if all the data will be in when trying this and if it would cause some other error makes things worse. Descriptions suggest another read would be needed, but don't know how at this level of the .get calls.

This is just some information around the error reported but there is no use of ssl logic in the code itself.

[catsmanac]

Persistent token cache has been added in V0.0.14 and v0.0.15 is now vaialble.

Token from cache is always used, unless expired. Communication to Enphase web site is only needed if token needs to be updated 1/year and during initial configuration to obtain first token.