Improve library and HA integration to reduce number of HTTP calls to Envoy device

[exxamalte]

I am using the envoy integration in Home Assistant for some time now and have an Envoy-S at home supporting production and consumption data, as well as 16 Inverters connected.

I have analysed the HTTP traffic between HA and the Envoy-S over a ~15 minute period. In that timeframe I saw:

• 462 calls to /production.json
• 432 calls to /info.xml
• 880 calls to /api/v1/production/inverters That's about 2 HTTP calls per second in average. From past experience (before this HA integration existed I used a rest sensor) I believe that the Envoy device is not really built for that amount of burst traffic.

Looking at the source code of both this library and the HA integration, I think the number of HTTP calls can be reduced significantly which in turn reduces network traffic and resource usage.

In the current version and in my hardware setup, the integration creates 24 sensors. Each sensor is updated once per minute, and in its async_update method it instantiates a new EnvoyReader and then calls a method to fetch data from the Envoy device. In case of production/consumption data, it always makes a call to detect the model, and then fetches all data and extracts the relevant data point for that sensor. In case of an inverter sensor it fetches all data and extracts the relevant data point.

I would propose to change the way how the library and integration work together. On a high level:

• Only detect the model once. This never changes.
• Only detect the serial number once. This never changes.
• Change the integration so that not each sensor must instantiate its own connection, but introduce a shared connection to this library which is updated once a minute and makes 2 calls - one to fetch production/consumption data and a second call to fetch inverter data. Then make this data internally available in HA and signal each of the sensors to update their state from that data pool.

This approach would reduce the number of calls from ~1700 to ~30 in that same time period.

I know that this change would require some significant work. Please let me know if you need any help.

[dalklein]

I agree, when making it work for my older Envoy-C, it seemed like it would be making many requests to the envoy, due to populating each individual value as it's own sensor. Although they are all pulled from the same page. It did work for my envoy, although mine does not offer pulling individual inverter data, so that is fewer requests to the envoy.

Additional thought: The data on the Envoy does not change at a very fast rate either, at least on the main summary page. 5 or 15 minutes, I forget. To be real fancy and efficient, start by checking every 1 minute. Each time that the data is still identical (envoy has not updated it's data yet), add another minute to the checking interval. And vice versa. Or something like that.

[exxamalte]

The data on the Envoy does not change at a very fast rate either, at least on the main summary page. 5 or 15 minutes, I forget.

My Envoy-S is updating its data every minute.

However, there may be an opportunity to reduce the update interval at night (say from ~1 hour after sunset until ~1 hour before sunrise), at least for inverters. The inverter data even says when it was last updated (lastReportDate).

[rct]

I just tried this on an Envoy-IQ, with HA 0.100.2 and the default where no monitored_conditions are specified.

While it seems to work, it overloads the Envoy with only 26 panels. My log has lots of exceptions for ReadTimeout and Updating enphase_envoy sensor took longer than the scheduled update interval 0:00:30

I'd love to be able to monitor at the panel level if the overhead could be reduced.

[rct]

Just started looking at this, first thing I noticed if I'm reading it correctly, is it looks like each attempt to get a value that HA uses like 'production', 'consumption', 'daily_consumption' calls self.call_api, which appears to fetch production.json even though it's probably already been fetched.

An easy fix without refactoring too much would be to just store the production parsed json object in the reader object. If it was successfully fetched recently enough to just use the already retrieved, parsed, json object.

[exxamalte]

An easy fix without refactoring too much would be to just store the production parsed json object in the reader object. If it was successfully fetched recently enough to just use the already retrieved, parsed, json object.

I would not recommend to start "fixing" the issue this way because this approach may introduce other complications. As you state you only want to reuse data if it "was successfully fetched recently enough". You may introduce unwanted side-effects.

The main reason I haven't started refactoring this myself is the lack of unit tests, i.e. you won't know for sure if this library still works as expected on all devices it supports.

[gtdiehl]

@exxamalte What tool did you use to monitor the HTTP traffic? Was it just Wireshark?

[exxamalte]

@gtdiehl: Yes, I used Wireshark to capture the traffic.

[gtdiehl]

@exxamalte Thank you.

I have started to try and reduce the number of calls that are made from Home Assistant to the Enphase Envoy device. I hacked together something that should reduce the number of calls drastically. Basically I have a single update() in the envoy_reader library that runs through all of the various methods, similar to what the run_in_console() is doing. Home Assistant will run the envoy_reader update() once every 30seconds (or whatever the user overrides the polling value to) and the sensors will get the data from Home Assistant. I felt this was the best approach for now, as the code on the envoy_reader library side is not touched, other than adding an update(). As most of the changes are on the Home Assistant side.

I'll try and commit my changes next week for everyone to take a look at.

[jesserizzo]

This is something I've been wanting to tackle for a while now, but don't have the time. Everyone here rocks.

[dalklein]

Yes, nice job, thanks for sharing!

[gtdiehl]

Here is a quick comparison I did using my Envoy running R3.11.30 with the current HA sensor code and the one I'm working on.

From the graphs the current sensor is making ~60 HTTP calls every polling period (default 30 seconds). Whereas with what I'm working on, the initial startup of HA server the calls are ~45, then every polling period after is reduced to ~10

More optimization can be done, but that would require more changes on the envoy_reader API side. Which is a larger task to make sure the changes don't break the various Envoys that are supported.

[gtdiehl]

Hi All! Here are the changes to the envoy_reader library and the Home Assistant Envoy sensor envoy_reader HA Envoy Sensor

I'm not going to make anymore to the envoy_reader library so I can minimize introducing any regressions or new bugs, but I'm still looking at what exceptions and stuff on the HA sensor side. Hoping to open a PR on that side in the next week or so.

[jesserizzo]

22 Closed

[gtdiehl]

Well it looks like the approach I took was not accepted on the HA side. I'll have to see about reorganizing the HA sensor code.

As well, it looks like HA 0.109 has introduced a Detected I/O message, and now it appears the inverter data gathering is now broken!

[ajmawer]

@gtdiehl would you be up for collaboration to see if we can get round 2 of this attempted? I have 32 micro’s so it tends to struggle - just need to get a dev setup running for HA to start poking around this....

[gtdiehl]

@ajmawer yeah let's try to fix this!

[gtdiehl]

@ajmawer @jesserizzo I have started to work on this again! I have pushed my initial HA sensor code to Enphase Envoy HA Sensor. Take a look and try it out.

Can this issue be re-opened or do I need to create a new one?

[rct]

@gtdiehl - Thanks for working on this.

I was just tracing envoy_reader running manually from the command line using your change_polling_pages branch. Any idea why the production/inverters page is hit twice?

GET /production.json HTTP/1.1
GET /api/v1/production HTTP/1.1
GET /info.xml HTTP/1.1
GET /api/v1/production/inverters HTTP/1.1
GET /api/v1/production/inverters HTTP/1.1

[gtdiehl]

@rct I looked at the modified code (actually the inverter production hasn't changed) and I don't see anything obvious as to why the inverter page would be retrieved twice. The only thing I can think of is the http communication is do to the authenticated page? Maybe two calls are made by the requests library one to authenticate and the other to retrieve the page?

[gtdiehl]

To update on this issue.

The PR on the Home Assistant side will reduce the number of calls made to the API and ultimately to the Envoy device. I made additional changes on the API side (that is on my branch as txt noted above).

When the HA changes are merged the API calls will be reduced some. Then when the API side changes are merged the calls to the Envoy device will be reduced even further. Just wanted to give a heads up it'll be a process but working towards it!

[rct]

The only thing I can think of is the http communication is do to the authenticated page? Maybe two calls are made by the requests library one to authenticate and the other to retrieve the page?

Yes, sorry you are right. It's doing digest auth, so it made an unauthenticated request, got the nonce, then was able to make the requested with the digest auth.

Tracing the current code running in Home Assistant, I get 54 HTTP requests to /api/v1/production/inverters per cycle for my 27 inverters.

Hopefully your new code once accepted will reduce that from 54 to 2 inverter requests per cycle.

[gtdiehl]

@rct No problem. What are you using to trace the http calls?

[rct]

In this case tcpflow since it is quick an easy to capture each request/response in files on a remote RPi.

[gtdiehl]

Okay thanks! I used tcpdump/tcptrace on just the Home Assistant envoy sensor.py changes. And it's inline with what I expected. Multiple calls being made to production pages (/api/v1/production & /production.json), but a reduction to only 2 calls made to /api/v1/production/inverters

    GET /production.json HTTP/1.1
    GET /production.json HTTP/1.1
    GET /production.json HTTP/1.1
    GET /production.json HTTP/1.1
    GET /production.json HTTP/1.1
    GET /production.json HTTP/1.1
    GET /production.json HTTP/1.1
    GET /production.json HTTP/1.1
    GET /api/v1/production/inverters HTTP/1.1
    GET /api/v1/production/inverters HTTP/1.1
    GET /api/v1/production HTTP/1.1
    GET /api/v1/production HTTP/1.1
    GET /api/v1/production HTTP/1.1
    GET /api/v1/production HTTP/1.1
    GET /api/v1/production HTTP/1.1
    GET /api/v1/production HTTP/1.1
    GET /api/v1/production HTTP/1.1
    GET /api/v1/production HTTP/1.1
    GET /api/v1/production HTTP/1.1
    GET /api/v1/production HTTP/1.1
    GET /api/v1/production HTTP/1.1
    GET /api/v1/production HTTP/1.1

The following reductions to /api/v1/production & /production.json should match your tcpflow output if/when the envoy_reader.py changes are merged from my change_polling_pages and Home Assistant bumps the envoy_reader version

I was using WireShark, but this way seems a bit quicker! Now I have to figure out why installing tcpflow killed my python environment! ;)

EDIT: Figured out why my environment died. tcpflow installed python3.9 and changed the symbolic link to python3 from 3.8 to 3.9. Anyway using tcpflow I can see that maybe the HA sensor.py code might still need some optimization, as what is seen above in the tcp capture is upon startup of HA but after startup on subsequent polls the production pages aren't hit as many times. But still this is a move in the right direction. Just have to keep iterating on making improvements! Thanks @rct for your help!

[rct]

@gtdiehl - re: tcpflow - Sorry for the trouble. What system are you on? I'm surprised tcpflow has any Python dependencies. The one I'm using is C/C++ based and it is pretty old utility On raspbian the package is 226k, 1 binary, a couple of doc files, and a manpage. The dependencies are:

Depends: libc6 (>= 2.8), libcairo2 (>= 1.2.4), libexpat1 (>= 2.0.1), libfontconfig1 (>= 2.12.6), libfreetype6 (>= 2.2.1), libgcc1 (>= 1:3.5), libhttp-parser2.8 (>= 2.1), libpcap0.8 (>= 0.9.8), libpixman-1-0, libssl1.1 (>= 1.1.0), libstdc++6 (>= 5.2), zlib1g (>= 1:1.1.4)

Back to the topic at hand, above you show 8 fetches of production.jsonand 12 fetches for /api/v1/production.

How many polling cycles does that represent?

[gtdiehl]

@rct Hey no problem. It was a quick fix. I'm running macOS and used brew to install tcpflow.

How many polling cycles does that represent?

One polling cycle at the startup of Home Assistant. After that the polling is reduced. This is what a subsequent polling cycle then looks like for an Envoy-S/IQ Envoy. Of course this is with the currently released envoy_reader API, so when the API is merged/updated the polling should look like what you captured with your tcpflow

GET /production.json HTTP/1.1
GET /api/v1/production/inverters HTTP/1.1
GET /production.json HTTP/1.1
GET /production.json HTTP/1.1
GET /production.json HTTP/1.1
GET /production.json HTTP/1.1
GET /production.json HTTP/1.1
GET /production.json HTTP/1.1
GET /production.json HTTP/1.1
GET /api/v1/production/inverters HTTP/1.1

[rct]

What polling (scan) interval are people using with Home Assistant?

Home Assistant's default is 30 seconds.

If you aren't trying to calculate WH yourself then probably only needs to be sampled every few minutes.

[gtdiehl]

@rct Looking through the Home Assistant envoy sensor.py code. I can see the scan_interval is set to 30 seconds in the sensor.py file

I'm thinking of adding a scan interval of every 60seconds. The envoy inverter data is updated every 5mins (15mins if you have older Envoy-C hardware) but the Watts value for production and consumption from a metered envoy appears to be real-time.

EDIT: I just updated the sensor.py and pushed my change for the same PR. The default scan_interval is 60 seconds and can be changed from the configuration.yaml using the scan_interval attribute for the envoy sensor.

[gtdiehl]

The changes on the Home Assistant side to reduce calls made to the Envoy will be in the 2021.1 release.

[gtdiehl]

Closing issue. API and Home Assistant changes have been made to reduce number of HTTP calls made to Envoy end point.