Serial number fetch for old Envoy crashes

[rianadon]

Since upgrading to the latest Home Assistant (with a new version of envoy_reader?), the enphase integration is failing.

I tried installing envoy_reader in a new environment, and it looks like there is an error fetching info.xml for retrieving the serial number, which causes getData to fail.

Running with HTTPX_LOG_LEVEL=debug:

>>> from envoy_reader.envoy_reader import EnvoyReader
>>> import asyncio
>>> e = EnvoyReader('envoy')
>>> asyncio.get_event_loop().run_until_complete(e.getData())
DEBUG [2021-06-10 12:16:46] httpx._client - HTTP Request: GET http://envoy/info.xml "HTTP/1.1 301 Moved Permanently"
DEBUG [2021-06-10 12:16:46] httpx._client - HTTP Request: GET http://envoy/info.xml "HTTP/1.1 301 Moved Permanently"
DEBUG [2021-06-10 12:16:46] httpx._client - HTTP Request: GET http://envoy/info.xml "HTTP/1.1 301 Moved Permanently"
Traceback (most recent call last):
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpx/_transports/default.py", line 61, in map_httpcore_exceptions
    yield
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpx/_transports/default.py", line 201, in __aiter__
    async for part in self._httpcore_stream:
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 57, in __aiter__
    async for chunk in self.stream:
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpcore/_bytestreams.py", line 91, in __aiter__
    async for chunk in self._aiterator:
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpcore/_async/http11.py", line 208, in _receive_response_data
    event = await self._receive_event(timeout)
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpcore/_async/http11.py", line 225, in _receive_event
    data = await self.socket.read(self.READ_NUM_BYTES, timeout)
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 65, in read
    raise ReadError("Server disconnected while attempting read") from None
httpcore.ReadError: Server disconnected while attempting read

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/envoy_reader/envoy_reader.py", line 135, in getData
    await self.detect_model()
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/envoy_reader/envoy_reader.py", line 165, in detect_model
    await self.get_serial_number()
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/envoy_reader/envoy_reader.py", line 218, in get_serial_number
    full_serial = await self.get_full_serial_number()
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/envoy_reader/envoy_reader.py", line 224, in get_full_serial_number
    response = await self._async_fetch_with_retry(
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/envoy_reader/envoy_reader.py", line 127, in _async_fetch_with_retry
    return await client.get(url, timeout=30, **kwargs)
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpx/_client.py", line 1661, in get
    return await self.request(
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpx/_client.py", line 1425, in request
    response = await self.send(
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpx/_client.py", line 1510, in send
    response = await self._send_handling_auth(
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpx/_client.py", line 1545, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpx/_client.py", line 1597, in _send_handling_redirects
    raise exc
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpx/_client.py", line 1590, in _send_handling_redirects
    await response.aread()
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpx/_models.py", line 1561, in aread
    self._content = b"".join([part async for part in self.aiter_bytes()])
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpx/_models.py", line 1561, in <listcomp>
    self._content = b"".join([part async for part in self.aiter_bytes()])
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpx/_models.py", line 1577, in aiter_bytes
    async for raw_bytes in self.aiter_raw():
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpx/_models.py", line 1631, in aiter_raw
    async for raw_stream_bytes in self.stream:
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpx/_client.py", line 127, in __aiter__
    async for chunk in self._stream:
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpx/_transports/default.py", line 202, in __aiter__
    yield part
  File "/opt/homebrew/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/ryan/Documents/venvs/test/lib/python3.9/site-packages/httpx/_transports/default.py", line 78, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ReadError: Server disconnected while attempting read
>>> 

[gtdiehl]

@rianadon What Envoy type do you have? And what is the firmware version? Thanks!

[rianadon]

Thanks for taking a look into this!

The firmware version is R3.7.31 (11b4e9).

Edit: I can't figure out what model it the gateway is, but it is from 2011. It's quite an old system 😁

[gtdiehl]

That's fine I don't need the type the firmware version is okay for now

[gtdiehl]

That is a strange error.

DEBUG [2021-06-10 12:16:46] httpx._client - HTTP Request: GET http://envoy/info.xml "HTTP/1.1 301 Moved Permanently"

I would expect a 200 return code for that page from an Envoy running D3.7.31

Does opening http://envoy directly in your browser show the Envoy webpage? What does http://envoy/info.xml show in a browser? Preferably from a browser on the same machine that is running Home Assistant if possible.

I ran the envoy_reader.py directly on an Envoy with the same version and I get valid results. Can you run envoy_reader.py? The username and password can be anything since this Envoy does not support Inverters. You should see some output similar to this:

(venv) greg@Gregs-iMac envoy_reader % python3 envoy_reader.py
Enter the Envoy IP address or host name, or press enter to use 'envoy' as default: x.x.x.x
Enter the Username for Inverter data authentication, or press enter to use 'envoy' as default: test
Enter the Password for Inverter data authentication, or press enter to use the default password: test
Reading...
production:              0
consumption:             Consumption data not available for your Envoy device.
daily_production:        517
daily_consumption:       Consumption data not available for your Envoy device.
seven_days_production:   10900
seven_days_consumption:  Consumption data not available for your Envoy device.
lifetime_production:     8390000
lifetime_consumption:    Consumption data not available for your Envoy device.
inverters_production:    Inverter data not available for your Envoy device.
battery_storage:         Battery storage data not available for your Envoy device.
(venv) greg@Gregs-iMac envoy_reader % 

Sorry for all the questions but I don't have another Home Assistant development environment right now. Tomorrow I'll load my Envoy into HA and see if I see the error you're getting. I don't expect to see an error as Home Assistant and envoy_reader.py are using the same function calls.

[rianadon]

http://envoy shows a System Overview website with a table containing lifetime generation, currently generating, etc.

http://envoy/info.xml redirects to http://envoy/error which looks like this:

envoy_reader.py works and gives similar output.

[rianadon]

This gets interesting. On my mac, the code works successfully. On the Linux VM I'm running Home Assistant, cloning envoy_reader and running envoy_reader.py gives (running with HTTPX_LOG_LEVEL=debug to see the HTTP requests):

Enter the Envoy IP address or host name, or press enter to use 'envoy' as default: 
Enter the Username for Inverter data authentication, or press enter to use 'envoy' as default: 
Enter the Password for Inverter data authentication, or press enter to use the default password: 
Reading...
DEBUG [2021-08-11 20:13:24] httpx._client - HTTP Request: GET http://envoy/info.xml "HTTP/1.1 301 Moved Permanently"
DEBUG [2021-08-11 20:13:24] httpx._client - HTTP Request: GET http://envoy/info.xml "HTTP/1.1 301 Moved Permanently"
DEBUG [2021-08-11 20:13:24] httpx._client - HTTP Request: GET http://envoy/info.xml "HTTP/1.1 301 Moved Permanently"
production:              local variable 'production' referenced before assignment
consumption:             'NoneType' object has no attribute 'json'
daily_production:        local variable 'daily_production' referenced before assignment
daily_consumption:       'NoneType' object has no attribute 'json'
seven_days_production:   local variable 'seven_days_production' referenced before assignment
seven_days_consumption:  'NoneType' object has no attribute 'json'
lifetime_production:     local variable 'lifetime_production' referenced before assignment
lifetime_consumption:    'NoneType' object has no attribute 'json'
inverters_production:    Inverter data not available for your Envoy device.
battery_storage:         'NoneType' object has no attribute 'json'

[rianadon]

Sorry for the comment spam. I made a new virtual environment on the Home Assistant machine, installed httpx, then ran envoy_reader.py, and the script was successful!

The problem is likely package versions.

Successful	Unsuccessful
``` anyio==3.3.0 certifi==2021.5.30 h11==0.12.0 httpcore==0.13.6 httpx==0.18.2 idna==3.2 rfc3986==1.5.0 sniffio==1.2.0 ```	``` anyio==3.1.0 certifi==2021.5.30 h11==0.12.0 httpcore==0.13.4 httpx==0.18.2 idna==2.10 rfc3986==1.4.0 sniffio==1.2.0 ```

I'll have to come back with this, but it looks like this isn't a problem with envoy_reader and instead a problem with whatever package versions Home Assistant has specified to use.

[gtdiehl]

No problem, that is interesting. It seems some underlying http library is not forwarding the request like it should.

In the code the allow_redirects is True, very strange.

    async def get_full_serial_number(self):
        """Method to get the  Envoy serial number."""
        response = await self._async_fetch_with_retry(
            "http://{}/info.xml".format(self.host),
            allow_redirects=True,
        )

I'll look into a bit more tomorrow in my development environment and see if I see the problem.

[rianadon]

I ran pip install --upgrade httpcore==0.13.6(that turned out to be the crucial package) in my Home Assistant venv and everything is up and working! Looks like maybe upgrading Home Assistant didn't upgrade all the packages.

Anyways, I think this was the commit in httpcore that fixed the Server disconnected while attempting read error: https://github.com/encode/httpcore/commit/10a5b0a2b0047f0c301b0bb551e43463592f54fe. So definitely not something from envoy_reader.

Thanks again for taking a look! Now it's time to play with the new energy dashboard 🤗