search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
71.55k stars 29.91k forks source link

Enphase Envoy contant disconnect from Envoy R #126162

Open luis-castro opened 1 day ago

luis-castro commented 1 day ago

The problem

My Envoy Enphase integration is configured with 2 gateways: 1 is running firmware D3.18.10 (Envoy R), and the other is running firmware 7.3.617 (Envoy IQ).

The integration is constantly losing data from the inverters connected to the Envoy R, showing them as Unavailable.

This is sometimes solved by restarting homeassist, others I have to delete the integration and add it again.

What version of Home Assistant Core has the issue?

core-2024.8.1

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

enphase_envoy

Link to integration documentation on our website

https://www.home-assistant.io/integrations/enphase_envoy

Diagnostics information

config_entry-enphase_envoy-01J815RY1A02ECBF94NKK8NKNV (1).json

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2024-09-17 17:48:30.551 ERROR (MainThread) [homeassistant.components.enphase_envoy.coordinator] Unexpected error fetching Envoy [*REDACTED S/N*] data
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
    yield
  File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 373, in handle_async_request
    resp = await self._pool.handle_async_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request
    raise exc from None
  File "/usr/local/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 196, in handle_async_request
    response = await connection.handle_async_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
    return await self._connection.handle_async_request(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpcore/_async/http11.py", line 143, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.12/site-packages/httpcore/_async/http11.py", line 113, in handle_async_request
    ) = await self._receive_response_headers(**kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpcore/_async/http11.py", line 186, in _receive_response_headers
    event = await self._receive_event(timeout=timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpcore/_async/http11.py", line 238, in _receive_event
    raise RemoteProtocolError(msg)
httpcore.RemoteProtocolError: Server disconnected without sending a response.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 354, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/enphase_envoy/coordinator.py", line 152, in _async_update_data
    envoy_data = await envoy.update()
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyenphase/envoy.py", line 443, in update
    await updater.update(data)
  File "/usr/local/lib/python3.12/site-packages/pyenphase/updaters/api_v1_production_inverters.py", line 40, in update
    inverters_data: list[dict[str, Any]] = await self._json_request(
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyenphase/updaters/base.py", line 34, in _json_request
    response = await self._request(end_point)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyenphase/envoy.py", line 384, in _make_cached_request
    response = await request_func(endpoint)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/tenacity/__init__.py", line 418, in exc_check
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/tenacity/__init__.py", line 185, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyenphase/envoy.py", line 216, in request
    return await self._request(endpoint, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyenphase/envoy.py", line 251, in _request
    response = await self._client.get(
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1801, in get
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1574, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1661, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1689, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1726, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpx/_client.py", line 1763, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 372, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/usr/local/lib/python3.12/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: Server disconnected without sending a response.

Additional information

No response

home-assistant[bot] commented 1 day ago

Hey there @bdraco, @cgarwood, @joostlek, @catsmanac, mind taking a look at this issue as it has been labeled with an integration (enphase_envoy) you are listed as a code owner for? Thanks!

Code owner commands Code owners of `enphase_envoy` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign enphase_envoy` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

(message by CodeOwnersMention)

enphase_envoy documentation enphase_envoy source (message by IssueLinks)

luis-castro commented 1 day ago

When I enabled DEBUG logs, it seems the integration is wrongly trying to use /production.json instead of the correct API /api/v1/production and therefore it is unable to decode the output (as expected, becuase the Envoy is returning an error).

If the integration has detected we are running firmware D3.18.10, then it should only make use of the following APIs:

2024-09-17 18:06:14.575 DEBUG (MainThread) [pyenphase.firmware] Request reply in 0.1 sec from http://192.168.1.15/info status 200: text/xml b"<?xml version='1.0' encoding='UTF-8'?>\n<envoy_info>\n  <time>1726617974</time>\n  <device>\n    <sn>[*REDACTED*]</sn>\n    <pn>800-00240-r03</pn>\n    <software>D3.18.10</software>\n    <euaid>4c8675</euaid>\n    <seqnum>0</seqnum>\n    <apiver>1</apiver>\n  </device>\n  <package name='backbone'>\n    <pn>500-00010-r01</pn>\n    <version>02.01.15</version>\n    <build>ad8746</build>\n  </package>\n  <package name='devimg'>\n    <pn>500-00004-r01</pn>\n    <version>01.02.245</version>\n    <build>033b78</build>\n  </package>\n  <package name='kernel'>\n    <pn>500-00003-r01</pn>\n    <version>02.04.00</version>\n    <build>80e63f</build>\n  </package>\n  <package name='security'>\n    <pn>500-00016-r01</pn>\n    <version>02.00.00</version>\n    <build>54a6dc</build>\n  </package>\n  <package name='boot'>\n    <pn>590-00015-r01</pn>\n    <version>02.00.01</version>\n    <build>0b54c5</build>\n  </package>\n  <package name='geo'>\n    <pn>500-00008-r01</pn>\n    <version>02.01.22</version>\n    <build>702db9</build>\n  </package>\n  <package name='app'>\n    <pn>500-00002-r01</pn>\n    <version>03.18.10</version>\n    <build>f0855e</build>\n  </package>\n  <package name='rootfs'>\n    <pn>500-00001-r01</pn>\n    <version>01.02.00</version>\n    <build>807</build>\n  </package>\n</envoy_info>\n"
2024-09-17 18:06:14.576 DEBUG (MainThread) [pyenphase.envoy] FW: 3.18.10, Authenticating to Envoy using envoy/installer authentication
2024-09-17 18:06:14.577 DEBUG (MainThread) [pyenphase.envoy] Requesting http://192.168.1.15/ivp/meters with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
2024-09-17 18:06:14.705 DEBUG (MainThread) [pyenphase.updaters.meters] Skipping meters endpoint as user does not have access to /ivp/meters: Authentication failed for http://192.168.1.15/ivp/meters with status 401, please check your username/password or token.
2024-09-17 18:06:14.706 DEBUG (MainThread) [pyenphase.envoy] Requesting http://192.168.1.15/production.json?details=1 with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
2024-09-17 18:06:14.933 DEBUG (MainThread) [pyenphase.envoy] Request reply in 0.2 sec from http://192.168.1.15/production.json?details=1 status 200: application/json b'    <!DOCTYPE html PUBLIC "-//IETF//DTD HTML 2.0//EN">\n    <html>\n    <!-- START HEAD CONTENT -->\n    <head>\n      <!-- include style sheet -->\n      <link rel="StyleSheet" type="text/css" href="/include/style.css" />\n      <!-- include JS libraries -->\n    </head>\n    <!-- END HEAD CONTENT -->\n    <body>\n    <center>\n    <img src="/images/emu-small.gif" style="margin:10px" alt="Emu"/>\n    <h2>Page Not Found</h2>\n    <p>The page you tried to view does not exist\n     Try the <a href="/">home</a> page</p>\n    </center>\n    </body>\n    </html>\n'
2024-09-17 18:06:14.934 DEBUG (MainThread) [pyenphase.json] Unable to decode response from Envoy endpoint /production.json?details=1: unexpected character: line 1 column 5 (char 4)
2024-09-17 18:06:14.934 DEBUG (MainThread) [pyenphase.updaters.production] Production endpoint not found at /production.json?details=1: unexpected character: line 1 column 5 (char 4)
2024-09-17 18:06:14.934 DEBUG (MainThread) [pyenphase.envoy] Requesting http://192.168.1.15/production with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
2024-09-17 18:06:15.082 DEBUG (MainThread) [pyenphase.envoy] Request reply in 0.1 sec from http://192.168.1.15/production status 200: application/json b'      <h1>System Energy Production</h1>\n      <div style="margin-right: auto; margin-left: auto;"><table>\n      <tr><td colspan="3">System has been live since\n                             <div class=good>Fri Sep 01, 2017 02:53 PM GMT+6</div></td></tr>\n      <tr><td>Currently</td>    <td>  49.0 W</td></tr><tr><td>Today</td>     <td> 3.27 kWh</td></tr><tr><td>Past Week</td>    <td> 26.9 kWh</td></tr><tr><td>Since Installation</td>    <td> 15.0 MWh</td></tr>\n      </table><br></div>\n'
2024-09-17 18:06:15.082 DEBUG (MainThread) [pyenphase.json] Unable to decode response from Envoy endpoint /production: unexpected character: line 1 column 7 (char 6)
2024-09-17 18:06:15.082 DEBUG (MainThread) [pyenphase.updaters.production] Production endpoint not found at /production: unexpected character: line 1 column 7 (char 6)
2024-09-17 18:06:15.082 DEBUG (MainThread) [pyenphase.envoy] Requesting http://192.168.1.15/api/v1/production with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
2024-09-17 18:06:15.116 DEBUG (MainThread) [pyenphase.envoy] Request reply in 0.0 sec from http://192.168.1.15/api/v1/production status 200: application/json b'{\n  "wattHoursToday": 3272,\n  "wattHoursSevenDays": 26896,\n  "wattHoursLifetime": 15043584,\n  "wattsNow": 49\n}\n'
2024-09-17 18:06:15.116 DEBUG (MainThread) [pyenphase.json] Unable to decode response from Envoy endpoint /production.json?details=1: unexpected character: line 1 column 5 (char 4)
2024-09-17 18:06:15.116 DEBUG (MainThread) [pyenphase.updaters.production] Production endpoint not found at /production.json?details=1: unexpected character: line 1 column 5 (char 4)
2024-09-17 18:06:15.117 DEBUG (MainThread) [pyenphase.envoy] Requesting http://192.168.1.15/api/v1/production/inverters with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
2024-09-17 18:06:16.801 DEBUG (MainThread) [pyenphase.envoy] Requesting http://192.168.1.15/api/v1/production/inverters with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
2024-09-17 18:06:19.201 DEBUG (MainThread) [pyenphase.envoy] Requesting http://192.168.1.15/api/v1/production/inverters with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
2024-09-17 18:06:20.839 DEBUG (MainThread) [pyenphase.envoy] Requesting http://192.168.1.15/api/v1/production/inverters with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
2024-09-17 18:06:20.879 DEBUG (MainThread) [pyenphase.updaters.api_v1_production_inverters] Production endpoint not found at /api/v1/production/inverters: Server disconnected without sending a response.
2024-09-17 18:06:20.879 DEBUG (MainThread) [pyenphase.updaters.ensemble] Firmware too old for Ensemble support
catsmanac commented 1 day ago

Hi @luis-castro, You are correct about the 2 API's it should use. Before I dive into the issue, I've got some question to assist with the troubleshooting.

The integration is constantly losing data from the inverters connected to the Envoy R, showing them as Unavailable.

Does it ever restore from unavailable without restarting Home-Assistant?

This is sometimes solved by restarting homeassist, others I have to delete the integration and add it again.

Looking at the information you provided (also see below) it seems as if the /api/v1/production/inverters is not reliably providing data. If this occurs during normal collection runs, the data will go to unavailable and come back when it works again. If this happens during a Home Assistant restart, the data will be considered as unavailable on the Envoy and not even be collected (some versions don't have this data).

If this happens you don't need to restart Home Assistant completely. If you open settings / devices & services / Enphase Envoy you can use the menu to reload the Envoy Integration. That will restart the communication and retry building all connections. This may work or not, but it avoids the whole restart of Home Assistant. And no, it's not a solution to the issue, just a simpler way to try to get it going again.

afbeelding

debug log

As for the debug log details, it shows is the startup sequence during which it determines what is available or not. This is not primary driving by the firmware version, but rather by the responses of the Envoy. This will include some failures depending on the Envoy model and firmware.

It received a successful reply for the /api/vi/production

2024-09-17 18:06:15.116 DEBUG (MainThread) [pyenphase.envoy] Request reply in 0.0 sec from http://192.168.1.15/api/v1/production status 200: application/json b'{\n  "wattHoursToday": 3272,\n  "wattHoursSevenDays": 26896,\n  "wattHoursLifetime": 15043584,\n  "wattsNow": 49\n}\n'

and as a result it will use this for production data.

The lines right below it is still part of the inspection and will lead to not trying to use the /production.json endpoint.

2024-09-17 18:06:15.116 DEBUG (MainThread) [pyenphase.json] Unable to decode response from Envoy endpoint /production.json?details=1: unexpected character: line 1 column 5 (char 4)
2024-09-17 18:06:15.116 DEBUG (MainThread) [pyenphase.updaters.production] Production endpoint not found at /production.json?details=1: unexpected character: line 1 column 5 (char 4)

The result is also reflected in the diagnostic file where both end-points are shown used. Raw_data is the data that was collected from the Envoy in the data last collection before the diagnostic report was created.

    "raw_data": {
      "/api/v1/production": {
        "wattHoursToday": 3260,
        "wattHoursSevenDays": 26887,
        "wattHoursLifetime": 15043572,
        "wattsNow": 46
      },
      "/api/v1/production/inverters": [
        {
          "serialNumber": "121726021970",
          "lastReportDate": 1726616739,
          "lastReportWatts": 11,
          "maxReportWatts": 218
        },
        {
          "serialNumber": "121726033042",

The next section trying to get the data from the inverters shows some issue though. No data is returned by the Envoy.

2024-09-17 18:06:15.117 DEBUG (MainThread) [pyenphase.envoy] Requesting http://192.168.1.15/api/v1/production/inverters with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
2024-09-17 18:06:16.801 DEBUG (MainThread) [pyenphase.envoy] Requesting http://192.168.1.15/api/v1/production/inverters with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
2024-09-17 18:06:19.201 DEBUG (MainThread) [pyenphase.envoy] Requesting http://192.168.1.15/api/v1/production/inverters with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
2024-09-17 18:06:20.839 DEBUG (MainThread) [pyenphase.envoy] Requesting http://192.168.1.15/api/v1/production/inverters with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
2024-09-17 18:06:20.879 DEBUG (MainThread) [pyenphase.updaters.api_v1_production_inverters] Production endpoint not found at /api/v1/production/inverters: Server disconnected without sending a response.

What can be seen is 4 connection attempts that fail right away and with varying wait time of 1-3 seconds in between. As a result the inverter data will probably not be collected after this restart and it would need another restart/reload on this Envoy in Home Assistant to try again.

The reported error Server disconnected without sending a response is also reported in the Error log of the original problem report. Although it's not clear if it was for the /api/v1/production/inverters endpoint as well. Based on your description it probably is.

luis-castro commented 17 hours ago

Hi @catsmanac,

Thanks for taking a look at it.

Answering your question: Yes, it has been working for at least a year. It started failing around 2 months ago.

I believe the issue is in the pyenphase library itself. I've been playing around and have the following findings:

  1. If I use curl to hit the API /api/v1/production/inverters, it works with no issues: 
    $ curl -u installer:[REDACTED] --digest http://192.168.1.15/api/v1/production/inverters
[
{
"serialNumber": "121726021970",
"lastReportDate": 1726672636,
"lastReportWatts": 177,
"maxReportWatts": 218
},
{
"serialNumber": "121726033042",
"lastReportDate": 1726672632,
"lastReportWatts": 186,
"maxReportWatts": 195
},
{
"serialNumber": "121726021253",
"lastReportDate": 1726672635,
"lastReportWatts": 85,
"maxReportWatts": 108
},
{
"serialNumber": "121730034753",
"lastReportDate": 1726672634,
"lastReportWatts": 190,
"maxReportWatts": 245
}
]

I can keep this repeating with no issues.

  1. If use the library I noticed that pyenphase/envoy.py is trying to probe all Updaters, even the ones we know won't work in firmare version 3.

  2. However, even if I comment out the Updaters, it seems the library is not using Digest Authentication when hitting the API /api/v1/production/inverters.

Here is the modification I applied to envoy,py:

UPDATERS: list[type["EnvoyUpdater"]] = [
#    EnvoyMetersUpdater,
#    EnvoyProductionJsonUpdater,
#    EnvoyProductionUpdater,
    EnvoyApiV1ProductionUpdater,
#    EnvoyProductionJsonFallbackUpdater,
    EnvoyApiV1ProductionInvertersUpdater,
#    EnvoyEnembleUpdater,
#    EnvoyTariffUpdater,
#    EnvoyGeneratorUpdater,
]

However, when I called envoy.update(), I got this:

DEBUG:pyenphase.envoy:Requesting http://192.168.1.15/api/v1/production/inverters with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_headers.failed exception=RemoteProtocolError('Server disconnected without sending a response.')
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete

It seems to me the library is not hitting the API with Digest Authentication (but I am not 100% certain of that).

For reference this is the Python script I am using for this test:

import asyncio
from pyenphase import Envoy, EnvoyData
from pprint import pprint
import logging

logging.basicConfig(level=logging.DEBUG)

async def main():
        envoy = Envoy("192.168.1.15")
        print('Calling envoy.setup()...')
        await envoy.setup()
        print(f'Envoy: {envoy.host}\nFirmare: {envoy.firmware}\nSerial: {envoy.serial_number}\n')

        print('Calling envoy.authenticate()...')
        await envoy.authenticate(username="installer")

        while True:
                print('Calling envoy.update()...')
                data: EnvoyData = await envoy.update()
                print(f'TodaysEnergy: {data.system_production.watt_hours_today}')
                for inverter in data.inverters:
                        print(f'{inverter} SN: {data.inverters[inverter].serial_number}')
                        print(f'{inverter} W: {data.inverters[inverter].last_report_watts}')
                print('Waiting 10 seconds...')
                await asyncio.sleep(10)

if __name__ == "__main__":
        print('Calling main...')
        asyncio.run(main())

What are your thoughts about this?

luis-castro commented 16 hours ago

It seems I was wrong about the library not using Digest Authentication, however something is not working.

Here is a complete dump of the output. I still don't understand what the problem is:

Calling envoy.authenticate()...
DEBUG:pyenphase.envoy:FW: 3.18.10, Authenticating to Envoy using envoy/installer authentication
Calling envoy.update()...
DEBUG:pyenphase.envoy:Requesting http://192.168.1.15/api/v1/production with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
DEBUG:httpcore.connection:connect_tcp.started host='192.168.1.15' port=80 local_address=None timeout=10.0 socket_options=None
DEBUG:httpcore.connection:connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x7fb621921990>
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Date', b'Wed, 18 Sep 2024 15:54:33 GMT'), (b'Content-Type', b'application/json'), (b'Content-Length', b'111')])
INFO:httpx:HTTP Request: GET http://192.168.1.15/api/v1/production "HTTP/1.1 200 OK"
DEBUG:httpcore.http11:receive_response_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_body.complete
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
DEBUG:pyenphase.envoy:Request reply in 0.0 sec from http://192.168.1.15/api/v1/production status 200: application/json b'{\n  "wattHoursToday": 889,\n  "wattHoursSevenDays": 27300,\n  "wattHoursLifetime": 15044482,\n  "wattsNow": 598\n}\n'
DEBUG:pyenphase.envoy:Requesting http://192.168.1.15/api/v1/production/inverters with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_headers.failed exception=RemoteProtocolError('Server disconnected without sending a response.')
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
DEBUG:pyenphase.envoy:Requesting http://192.168.1.15/api/v1/production/inverters with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
DEBUG:httpcore.connection:connect_tcp.started host='192.168.1.15' port=80 local_address=None timeout=10.0 socket_options=None
DEBUG:httpcore.connection:connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x7fb621974550>
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_headers.complete return_value=(b'HTTP/1.1', 401, b'Unauthorized', [(b'Date', b'Wed, 18 Sep 2024 15:54:34 GMT'), (b'WWW-Authenticate', b'Digest qop="auth", realm="enphaseenergy.com", nonce="1726674874"'), (b'Content-Type', b'application/json'), (b'Content-Length', b'90')])
INFO:httpx:HTTP Request: GET http://192.168.1.15/api/v1/production/inverters "HTTP/1.1 401 Unauthorized"
DEBUG:httpcore.http11:receive_response_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_body.complete
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_headers.failed exception=RemoteProtocolError('Server disconnected without sending a response.')
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
DEBUG:pyenphase.envoy:Requesting http://192.168.1.15/api/v1/production/inverters with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
DEBUG:httpcore.connection:connect_tcp.started host='192.168.1.15' port=80 local_address=None timeout=10.0 socket_options=None
DEBUG:httpcore.connection:connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x7fb621974f10>
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_headers.complete return_value=(b'HTTP/1.1', 401, b'Unauthorized', [(b'Date', b'Wed, 18 Sep 2024 15:54:36 GMT'), (b'WWW-Authenticate', b'Digest qop="auth", realm="enphaseenergy.com", nonce="1726674876"'), (b'Content-Type', b'application/json'), (b'Content-Length', b'90')])
INFO:httpx:HTTP Request: GET http://192.168.1.15/api/v1/production/inverters "HTTP/1.1 401 Unauthorized"
DEBUG:httpcore.http11:receive_response_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_body.complete
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_headers.failed exception=RemoteProtocolError('Server disconnected without sending a response.')
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
DEBUG:pyenphase.envoy:Requesting http://192.168.1.15/api/v1/production/inverters with timeout Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
DEBUG:httpcore.connection:connect_tcp.started host='192.168.1.15' port=80 local_address=None timeout=10.0 socket_options=None
DEBUG:httpcore.connection:connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x7fb621977110>
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_headers.complete return_value=(b'HTTP/1.1', 401, b'Unauthorized', [(b'Date', b'Wed, 18 Sep 2024 15:54:36 GMT'), (b'WWW-Authenticate', b'Digest qop="auth", realm="enphaseenergy.com", nonce="1726674876"'), (b'Content-Type', b'application/json'), (b'Content-Length', b'90')])
INFO:httpx:HTTP Request: GET http://192.168.1.15/api/v1/production/inverters "HTTP/1.1 401 Unauthorized"
DEBUG:httpcore.http11:receive_response_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_body.complete
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'GET']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'GET']>
DEBUG:httpcore.http11:receive_response_headers.failed exception=RemoteProtocolError('Server disconnected without sending a response.')
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete
DEBUG:pyenphase.updaters.api_v1_production_inverters:Production endpoint not found at /api/v1/production/inverters: Server disconnected without sending a response.
TodaysEnergy: 889
Waiting 10 seconds...

BTW: As this is not a home-assistant issue per se, let me know if I should file an issue with the pyenvoy team.

catsmanac commented 16 hours ago

So your test program shows the issue as well. In the end it's probably a pyenhase library issue, before we move there I want to verify some things. I'm member of the pyenphase team as well, so we can switch when needed.

In your test code can you change start of main to:


from pyenphase import register_updater
from pyenphase.envoy import get_updaters
from pyenphase.updaters.base import EnvoyUpdater
from pyenphase.updaters.production import EnvoyProductionJsonUpdater, EnvoyProductionUpdater, EnvoyProductionJsonFallbackUpdater

logging.basicConfig(level=logging.DEBUG)

async def main():
    updaters: list[type[EnvoyUpdater]] = get_updaters()
    if EnvoyProductionJsonUpdater in updaters:
        _LOGGER.debug("Removing EnvoyProductionJsonUpdater from Pyenphase")
        updaters.remove(EnvoyProductionJsonUpdater)
    if EnvoyProductionUpdater in updaters:
        _LOGGER.debug("Removing EnvoyProductionUpdater from Pyenphase")
        updaters.remove(EnvoyProductionUpdater)
    if EnvoyProductionJsonFallbackUpdater in updaters:
        _LOGGER.debug("Removing EnvoyProductionJsonFallbackUpdater from Pyenphase")
        updaters.remove(EnvoyProductionJsonFallbackUpdater)
    envoy = Envoy("192.168.1.15")
    print('Calling envoy.setup()...')

This will remove the production endpoint updaters and prevent reading the production pages at startup probe to see if that removes the issue or not.

Let it start and run for 2 or 3 cycles and then upload the debug log here.

luis-castro commented 15 hours ago

Done.

BTW: I got an error with _LOGGER.debug, so I changed it to print (sorry, I'm new to Python!).

Here is the output: pytest.log

catsmanac commented 15 hours ago

Can try with username='envoy' as well?

luis-castro commented 15 hours ago

Sure, here is the output, but there is no change: output-envoy.log

Anyway, I think "installer" should work as per my tests with curl.

catsmanac commented 14 hours ago

Anyway, I think "installer" should work as per my tests with curl.

Agree

Ok, well not ok yet, on to next step. Let's eliminate the meters and tariff collection. Should only leave info, v1/production and inverters for collection.

Add 

from pyenphase.updaters.meters import  EnvoyMetersUpdater
from pyenphase.updaters.tariff import EnvoyTariffUpdater

and

if EnvoyMetersUpdater in updaters:
    _LOGGER.debug("Removing EnvoyMetersUpdater from Pyenphase")
    updaters.remove(EnvoyMetersUpdater)
if EnvoyTariffUpdater in updaters:
    _LOGGER.debug("Removing EnvoyTariffUpdater from Pyenphase")
    updaters.remove(EnvoyTariffUpdater)

right after the already added sections

luis-castro commented 13 hours ago

Applied the modifications and ran the new test.

See attached log: test-20240918-1925UTC.txt

For reference, this is how the test code looks now:

import asyncio
from pyenphase import Envoy, EnvoyData
from pyenphase import register_updater
from pyenphase.envoy import get_updaters
from pyenphase.updaters.base import EnvoyUpdater
from pyenphase.updaters.production import EnvoyProductionJsonUpdater, EnvoyProductionUpdater, EnvoyProductionJsonFallbackUpdater
from pyenphase.updaters.meters import EnvoyMetersUpdater
from pyenphase.updaters.tariff import EnvoyTariffUpdater
import logging

logging.basicConfig(level=logging.DEBUG)

async def main():
        updaters: list[type[EnvoyUpdater]] = get_updaters()
        if EnvoyProductionJsonUpdater in updaters:
                logging.debug("Removing EnvoyProductionJsonUpdater from Pyenphase")
                updaters.remove(EnvoyProductionJsonUpdater)
        if EnvoyProductionUpdater in updaters:
                logging.debug("Removing EnvoyProductionUpdater from Pyenphase")
                updaters.remove(EnvoyProductionUpdater)
        if EnvoyProductionJsonFallbackUpdater in updaters:
                logging.debug("Removing EnvoyProductionJsonFallbackUpdater from Pyenphase")
                updaters.remove(EnvoyProductionJsonFallbackUpdater)
        if EnvoyMetersUpdater in updaters:
                logging.debug("Removing EnvoyMetersUpdater from Pyenphase")
                updaters.remove(EnvoyMetersUpdater)
        if EnvoyTariffUpdater in updaters:
                logging.debug("Removing EnvoyTariffUpdater from Pyenphase")
                updaters.remove(EnvoyTariffUpdater)

        envoy = Envoy("192.168.1.15")
        logging.debug('Calling envoy.setup()...')
        await envoy.setup()
        print(f'Envoy: {envoy.host}\nFirmare: {envoy.firmware}\nSerial: {envoy.serial_number}\n')

        logging.debug('Calling envoy.authenticate()...')
        await envoy.authenticate(username="installer")

        while True:
                logging.debug('Calling envoy.update()...')
                data: EnvoyData = await envoy.update()

                print(f'TodaysEnergy: {data.system_production.watt_hours_today}')

                for inverter in data.inverters:
                        print(f'{inverter} SN: {data.inverters[inverter].serial_number}')
                        print(f'{inverter} W: {data.inverters[inverter].last_report_watts}')

                print('Waiting 10 seconds...')
                await asyncio.sleep(10)

if __name__ == "__main__":
        logging.debug('Calling main...')
        asyncio.run(main())
luis-castro commented 13 hours ago

I thought the Envoy was blacklisting my IP, but I ran curl from the same machine I'm running python and it works:

$ curl -u installer:REDACTED --digest http://192.168.1.15/api/v1/production/inverters
[
  {
    "serialNumber": "121726021970",
    "lastReportDate": 1726687061,
    "lastReportWatts": 124,
    "maxReportWatts": 218
  },
...
catsmanac commented 12 hours ago

So behavior is pretty consistent with access denied on the inverters endpoint.

Answering your question: Yes, it has been working for at least a year. It started failing around 2 months ago.

Any idea what version upgrade that was? I.e. if you made any backup of home assistant before upgrades you can see version of the backup in you backup list.

I'll do some search in what all we changed in pyenphase around that time, since like 3 months ago. That will be tomorrow as hours in the day are running out quickly here now.

luis-castro commented 12 hours ago

Unfortunately the earliest backup I keep is from August 24. Back then I was running Home Assistant 2024.8.1 (same version I'm currently running), but I think I applied updates to some add-ons.

I appreciate all your efforts! Let me know if more information or tests are needed.

If you need access to this Envoy, I can arrange that too.

luis-castro commented 10 hours ago

I have reviewed my statistics, the issue I had two months ago was unrelated (sorry about that!)

As you can see in this screenshot, the connectivity became unrealiable since September 6: Screenshot 2024-09-18 16 30 11

Today I upgraded Home Assistant to 2024.9.2 and upgraded Home Assistant OS to version 13.1. The inverters were showing results for a couple of minutes and then they became unavailable again.