briancmpbll / home_assistant_custom_envoy

176 stars 77 forks source link

Inverter update frequency now once per hour (?) #201

Closed greggitter closed 1 month ago

greggitter commented 1 month ago

Looks like something changed, whereas I was getting individual inverter power updates about every 5 minutes, that's now shifted to one update per hour. I use this to track the percent contribution of each of my three arrays. In the past this was actually useful as I had a couple inverters offline and was able to identify that quickly, contacting Enphase and them identifying the problem and solving remotely. Here's a look at when the update frequency changed. I'm curious if anyone else seeing this? I reached out to Enphase to see if they have any info on this. BTW, I manually opened the local api page to verify that was the source of the issue.

image

catsmanac commented 1 month ago

Hi @greggitter, any idea if anything happened around the time of the red circle? HA update, HA restart, power outage.......

The first day on the left in the screenshot is drawn from state data, while on the right of the red circle the data seems to be drawn from statistical data, the 1 hour intervals you refer too. Typically it's the other way around as state data is only stored for a limited number of days. So this could be something related to your HA internals as well (https://community.home-assistant.io/t/psa-2024-7-recorder-problems/746428).

To start with a simple check to validate if this is an envoy issue or not, first enable debug on the Envoy integration. Let it run for some 30 min and disable it again. The log file will download to your device. In the log file you can find the raw data that came from the envoy. Look for log entries like below. as you mentioned, individual entries should update every 5 minutes. And each inverter will probably change at a different moment in time

2024-06-06 08:08:03.049 DEBUG (MainThread) [custom_components.enphase_envoy.envoy_reader] Fetched (1 of 2) in 0.2 sec from https://10.1.6.126/api/v1/production/inverters: <Response [200 OK]>: [
  {
    "serialNumber": "987654321012",
    "lastReportDate": 1717657462,
    "devType": 1,
    "lastReportWatts": 99,
    "maxReportWatts": 294
  },
  {
    "serialNumber": "987654321013",
    "lastReportDate": 1717657013,
    "devType": 1,
    "lastReportWatts": 101,
    "maxReportWatts": 292
  },
  {

If no changes are visible resume the debug log to span at least 2 hours and check again.

greggitter commented 1 month ago

No, no power anomalies or anything...I verified the api inverter page and it is definitely the source of the aged data. Nothing wrong with HA. Just wondered if anyone else is now seeing this. Haven't heard back from Enphase yet. I think I'll try rebooting the Envoy tonight after the sun sets (or just before). Thx.

catsmanac commented 1 month ago

I think I'll try rebooting the Envoy tonight after the sun sets (or just before).

Rebooting the envoy would be my next suggestion indeed.

So the inverter data is stale, but HA communication with the Envoy is just working fine?

greggitter commented 1 month ago

I think I'll try rebooting the Envoy tonight after the sun sets (or just before).

Rebooting the envoy would be my next suggestion indeed.

So the inverter data is stale, but HA communication with the Envoy is just working fine?

Yes, that's what I'm seeing...HA is updating my dashboard every 15 seconds with correct data regarding production and load, which is obviously more important for history tracking. Verifying the enphase api web page for inverters...that data appears stale and basically the same data HA retrieved.

An example of some inverters from the API now shows as below (local time is 10:13 AM); I'm not sure what format the timestamp is in, but the reported watts are closer to the 295 max number at the moment (direct sun) whereas HA is reporting one hour ago (stale) but matching the API data:


  {
    "serialNumber": "202245078365",
    "lastReportDate": 1722172923,
    "devType": 1,
    "lastReportWatts": 188,
    "maxReportWatts": 298
  },
  {
    "serialNumber": "202245088446",
    "lastReportDate": 1722172895,
    "devType": 1,
    "lastReportWatts": 192,
    "maxReportWatts": 298
  },
  {
    "serialNumber": "202245080469",
    "lastReportDate": 1722173044,
    "devType": 1,
    "lastReportWatts": 194,
    "maxReportWatts": 297
  },
  {
    "serialNumber": "202245086566",
    "lastReportDate": 1722172896,
    "devType": 1,
    "lastReportWatts": 193,
    "maxReportWatts": 299
  },
  {
    "serialNumber": "202245086568",
    "lastReportDate": 1722172924,
    "devType": 1,
    "lastReportWatts": 194,
    "maxReportWatts": 298
  },
  {
    "serialNumber": "202245088408",
    "lastReportDate": 1722172925,
    "devType": 1,
    "lastReportWatts": 194,
    "maxReportWatts": 299
  },```

EDIT: It just updated and 250w is the real time number.
catsmanac commented 1 month ago

The timestamp is seconds since 1 January 1970 The first timestamp 1722172923 is 2024-07-28 13:22:03 in UTC. The second timestamp is 28 seconds earlier (2024-07-28 13:21:35), the third one 121 seconds later (2024-07-28 13:24:04) and so on.

If your local time is 10:13 and you are in utc-4 (based on the github email time) then the timestamp would be 09:22:03 am, so almost 50 min stale. Really curious indeed.

greggitter commented 1 month ago

@catsmanac thanks for deciphering and explaining.

EDIT: Rebooting Envoy had no effect.

greggitter commented 1 month ago

Spoke to enphase support, here's the summary of a 45+ minute call.

Site is fully operational and no issues found however we did discover that the API settings were changed to 1 hour reporting from five min and I have submitted a email to fix the issue.

It's been escalated to a different support team which can take up to 3 days but probably sooner I'm told. I'll update this post once resolved and close this thread.

catsmanac commented 1 month ago

Thanks for the feedback @greggitter.

greggitter commented 1 month ago

I had to call Enphase again and after an hour the tech said he'd call back. I didn't expect this to get this resolved because this isn't something they deal with ever (at least this individual). He did call back and reported it resolved. There is a setting in the Enlighten Manager (on their end) called Polling Configuration and somehow, mine was changed on the 26th to 3600. The tech changed it back to 300 and it is operating per spec.

Documentation....

Get inverter production data

GET http://{IQ_Gateway_ip}/api/v1/production/inverters Description: Returns the available microinverters' maximum and last reported active power production information. This data will get updated once every five minutes.

https://enphase.com/download/accessing-iq-gateway-local-apis-or-local-ui-token-based-authentication (page 19)

chairstacker commented 4 weeks ago

Just a quick 'Thank you!' for the help here and a quick comment: In the last 10 days the pooling value was set back from 300 to 3600 twice again - I contacted Enphase Support about it via chat and they fixed it twice. I was told that their Support Team is aware of the issue with the value changing back to 3600 automatically every few days and they’re working on a solution.

greggitter commented 4 weeks ago

Just a quick 'Thank you!' for the help here and a quick comment: In the last 10 days the pooling value was set back from 300 to 3600 twice again - I contacted Enphase Support about it via chat and they fixed it twice. I was told that their Support Team is aware of the issue with the value changing back to 3600 automatically every few days and they’re working on a solution.

I too had mine revert after the first fix from 3600 back to 300...so also 2x here. Yep it was about three days after that initial fix though I'm on day six now and still good. Appreciate the feedback and good to know it wasn't something specific to my system/account. Still wonder what purpose it serves to change it at all...? Cheers.

chairstacker commented 4 weeks ago

FYI: I activated the 'Last Reported' sensor for one of my micro inverters and set up an automation in Home Assistant to notify me if the value doesn't change for 15min:

trigger:
  - platform: template
    value_template: >-
      {{ (as_timestamp(now()) - as_timestamp(states('sensor.inverter_482252022420_last_reported'))) > 900  }}
greggitter commented 4 weeks ago

@chairstacker I'll give that a try as well. Good idea, thanks. 👍

catsmanac commented 4 weeks ago

How about that for a new feature. Thanks for sharing the experience and the automation.

greggitter commented 4 weeks ago

@chairstacker Coincidentally while adding this, I noticed my polling interval has reverted AGAIN away from 5 minutes. I emailed my contact at enphase.

catsmanac commented 4 weeks ago

What firmware versions do you have on your Envoy? Wonder if there's a relation there.

greggitter commented 4 weeks ago

Gateway is D7.3.621. They attempted to update it but it failed, for unknown reasons. Just as well as this has been working fine and I don't want to be incompatible with HA.

chairstacker commented 4 weeks ago

Same firmware version on my gateway - not sure if an effort to upgrade was made. I don't think it's a gateway issue, though.

Looking at the history in more detail I saw that the issue showed the first time on July 26th around 6:15AM MST. I was out of town and only found this thread around Aug 10th when it was fixed .... but it had 'healed itself' between Aug 3rd and Aug 8th (polling interval 300) when the polling interval seemed to go back to 3600 at around 11AM MST.
On Aug 14th it went back to 3600 at around 8:30AM MST, got fixed the same day and lasted until today Aug 20th at 8:45AM MST.

These are the setting I received after the Aug 14 fix:

Reporting/Polling/Binning Configuration
Server (report) reports.enphaseenergy.com
Server (end_user) reports.enphaseenergy.com
Server (ping) ping-udp.enphaseenergy.com
Server (secure) reports.enphaseenergy.com
Reporting Interval (ping) 300 secs
Reporting Interval (interval) 300 secs
Reporting Interval (event) 300 secs
Reporting Interval (reading) 300 secs
Polling Period (pcu enabled) 300 secs
Polling Period (rgm enabled) 300 secs
Polling Period (default ) 300 secs
Bin Width (pcu) 300 secs
Bin Width (pmu) 300 secs
Bin Width (cmu) 300 secs
Bin Width (environ) 300 secs
Bin Width (rgm) 900 secs
Bin Width (default) 300 secs
Reporting Bandwidth high 
greggitter commented 4 weeks ago

Interesting, seems they have something going on there in an automated fashion. Are you using wifi? How frequently are you querying from HA? I'm set at 15 second intervals...wi-fi but have a cable here as I've been wanting to throw it on ethernet...not that my wifi signal is bad, just that I prefer wired whenever possible. The one advantage of wifi is the buffer to my homelab in the case of a power surge...nothing hard wired on the data side.

How did you obtain those settings?

chairstacker commented 4 weeks ago

My gateway is on WiFi, too - I have one LAN connection to the area where all my solar kit is, and it's used by the Tesla Gateway because that's (even) less reliable that the Enphase WiFi connection.

I save the stats in HA every 5min which lines up nicely with the 300sec interval I see them reporting on the Enphase website: image

image

BTW, the 'flat' section after 8:40AM is when the polling interval was changed to 3600 (fixed by 9:50AM) and at 10ish the panel was in full production mode at 298W.

The support tech sent the changes via chat - I'll keep them and paste them back into the chat whenever I need to show what the target setup should be 😉

You don't, by any chance, have a Powerwall2 in your system, do you?

greggitter commented 4 weeks ago

OK, I do that as well with regards to the inverters...I think that was automatic when I added it to HA. I have three arrays and calculate the percent production on each so if there's a problem I can identify it quickly (and to have a few more gauges on the dashboard). That happened twice where I had a few inverters that were not producing because my line voltage was too high at a little over 250v. A grid issue. They checked and changed my grid profile to be compatible with a higher voltage, closer to 260 (which I'm told is the max). The grid later fixed itself but here's what it was doing.

image

Neat...huh?

From HA how often do you query your gateway? I was wondering if maybe the gateway is complaining that is having trouble keeping up even though I had no issues for more than a year. My system is 18 months old, though I did just add 5 additional panels at the end of May giving me a total of 24. I query every 15 seconds for the load and production numbers to use on the energy dashboard and for long term stats and to reconcile the electric bill (and Enphase data).

I'll try a chat if my email doesn't get acknowledged tomorrow...those settings will prove handy in that case.

chairstacker commented 4 weeks ago

Wow - that's wild 😮

My Enphase sensors with the power production and consumption values seem to update every 62ish seconds, but I can't use the consumption values because they include the power that goes to the Tesla battery for charging as well, so I need to use the Tesla consumption ones.
And I found out that I can't really use the Tesla production values because they are between 7% and 8% too high. It's a mess 😕

Not that I need to increase the database size, but is there a way to increase the frequency of retrieving the Enphase values?

greggitter commented 4 weeks ago

Interesting. To change the update frequency go to the enphase service and click configure…several options there. Also they reset my interval again to 300 and said they would look into this more tomorrow.

chairstacker commented 4 weeks ago

Sorry, I'm not following. Is that on the Enphase website itself (https://enlighten.enphaseenergy.com/) or on the local gateway? Can't see this kind of option in either place, though.

greggitter commented 4 weeks ago

Sorry…in home assistant settings devices and services…select the enphase service (addon) and then choose configure. Unless I misunderstood what you want to do. I set my refresh to 15 seconds…the first option.

chairstacker commented 4 weeks ago

Looks like that option has recently been removed and one needs to set up a custom polling interval - not sure I want to go through that trouble (at least not just yet): https://www.home-assistant.io/integrations/enphase_envoy#polling-interval

Polling interval The default polling interval is 60 seconds. To customize the polling interval, refer to defining a custom polling interval. Specify one single entity from the envoy device as target of the action using the + choose entity button. Updating one entity will update all entities of the Envoy and the related devices like the inverters; there is no need to specify multiple or all entities or add (all) inverter entities. When using multiple Envoys, add one entity for each envoy as targets or create separate custom polling intervals with a single entity as needed.

catsmanac commented 4 weeks ago

Ah, yes if you run the HA core integration. This site is for a custom integration that has that option. I referred you to this one as the issue was an Enphase/Envoy issue, not a integration issue and was mentioned here.

(I'm providing support on both this one based om my past participation and on the core one when migrated function from this to the core one.)

To set your custom interval for the core integration that link has the description.

greggitter commented 4 weeks ago

@catsmanac Thanks for clearing that up...I probably would have missed that since as you said, this site is for the custom integration. :) And thanks for maintaining this...it's rock solid. Cheers!

chairstacker commented 4 weeks ago

Sorry @catsmanac - completely missed that this is for the custom integration. Totally appreciate your help!

catsmanac commented 4 weeks ago

Good to see were all helping out each other, asking and sharing helps us all learning.

chairstacker commented 3 weeks ago

Yup, and me not using the custom integration convinces me even more that it's a system side/backend issue with Enphase.

greggitter commented 3 weeks ago

Just updating this issue continues every few days it's being reset to 3600 second interval.

Edit: I called to have it fixed, they didn't fix it after 40 minutes. I opened a chat and it was fixed almost immediately...but only lasted about 4 hours. I opened a chat again and had it fixed again...sigh.

chairstacker commented 3 weeks ago

Mine just got reversed to 3600 again as well - still trying to get a slot with their chat support - sigh ☹️ Looks like it actually happened yesterday afternoon already, but my automation didn't trigger because of ... my own fault.

greggitter commented 3 weeks ago

Just an FYI....the guy I spoke with yesterday gave me a link to contact the API team about this. I'm not sure why this is an API issue, but rather a configuration issue. I think they continue to be confused when I tell them I'm using the token-based API to the gateway to get inverter status. I went ahead and used the link but all it does is takes the information entered on their page and throws into an unformatted, difficult to read email (my email client is thunderbird on my computer). I was annoyed and just sent it. Here's another interesting tidbit...they must be blocking the domain "outlook.com" with a spam filter (never gets there) so if you ever contact them via email, use something else. Gmail seems to work fine.

https://developer-v4.enphase.com/docs/support

catsmanac commented 3 weeks ago

So the Saga continues....

chairstacker commented 3 weeks ago

Another day, another Enphase reset - polling interval seems to be back to 3600 just about 35min ago 😕 Not prepared yet to follow the advice I received last time (3 days ago) and ask for it to be set to 900 instead of 300 because 'ideally it should be 900 secs as the system reports every 15 mins' 😮

greggitter commented 3 weeks ago

That's a lame reason, and that's disappointing but I don't think most of the techs understand the token based API and how it's useful. It just seems like there must be someone higher up the chain that can get involved with a fix or something. I emailed my contact yesterday explaining it happened for me twice on Saturday. Prior, he said he was going to find out what's going but he didn't get back to me. The documentation says 5 minutes. It's been fine for nearly 18 months of me using it and it only started a month ago. Something changed. And it can't be a wi-fi/bandwidth issue since the reporting is between micros and gateway. So what problem would it solve for them to change the interval to one hour...I can't think of any.

chairstacker commented 3 weeks ago

This is what I'm told every time I get through: "due to some patch updates the polling interval has changed." I have asked for a contact in the backend team to help me understand WHY this keep happening: crickets! Can you please point me to the documentation where it refers to the 5min/300sec interval? I'll then bring it up with the support folks in the future.

greggitter commented 3 weeks ago

Yeah I linked the document above. My contact has been responsive until this last note, of course he could be out of the office too.

Edit: My inverters just reset again too...sigh.

catsmanac commented 3 weeks ago

"due to some patch updates the polling interval has changed."

My envoy (EU standard not metered) received the latest patch 8.2.4264, came from 7.6.175, last week (driven by reported vulnerabilities) and it is updating every 300 sec as always.

The other big change is the new AI Optimization mode for batteries which requires that 8.2.42xx firmware to allow for battery energy sell/buy. That probably came with changed behavior on the cloud side. But that's me speculating. But since your Envoy firmware didn't change I wonder if it is some command send by the cloud when the Envoy communicates the updates and checks for commands.

greggitter commented 3 weeks ago

@chairstacker I can't get chat to work on desktop ever...works fine on mobile. Was going to open a ticket for that too but thought I'd check to see if it was working for you. Their website allows you to change your email but when you hit save and then refresh, it reverts...so that too looks broken.

chairstacker commented 3 weeks ago

@greggitter - chat from the desktop works fine for me; it's actually my preferred method of contacting Enphase because I'm not tethered to the phone then and I can just paste in my info and the whole spiel I've developed by now. I just always have to go to the 'Enphase Community' site (https://support.enphase.com/s/contact-us) first and then scroll down before the '... Chat With An Agent' thingie appears at the bottom right corner of the screen. After this point it's just watching the countdown for my place in line - it's started as high as 29, but normally moves rather swiftly 👍 Haven't had the need to change the email address - so, I do not have any experience with that.

greggitter commented 3 weeks ago

OK, THAT seems to work. Would you mind trying by starting on your status page, and then clicking the phone icon in the top right, then click chat with us...? If you can't access it I'll open a ticket for that but will start using the other route near-term. That same method that doesn't work on my desktop works fine on mobile...so....just want to confirm it's not me. Thx.

chairstacker commented 3 weeks ago

@greggitter - I assume when you say 'on your status page' you're referring to the mobile app, correct?

If I use the Enphase app and tap on the phone icon, I get two options:

  1. Contact Enphase
  2. Contact Maintainer

Under 'Contact Enphase' I'm offered these options: image

'Chat with us' opens a form that has my info pre-filled (and I can't adjust anything other than the 'Subject' field) and then sends me to the queue.

Is that what you're looking for? Seems to work all as intended for me 😕

greggitter commented 3 weeks ago

No, desktop, your account status where it shows production, consumption, import/export. Mobile works fine for me...the desktop webpage does not unless I use the route you suggested.

image

chairstacker commented 3 weeks ago

When I click on that icon, I get to a page that offers these options with 'Contact Us' pre-selected:

image

And clicking on 'Chat With Us' takes me to the pre-filled form with my contact and Sited ID info and on to the same queue as my original option - just in a different color 😮

greggitter commented 3 weeks ago

OK, I don't get the pre-filled form, just a spinner. So it's my desktop somehow...I'll look into it. Thanks @chairstacker .

chairstacker commented 4 days ago

Just a quick update that I have NOT had a reporting/API issue since Aug 27th - I hope I'm not jinxing it right now 😕

greggitter commented 2 days ago

Just a quick update that I have NOT had a reporting/API issue since Aug 27th - I hope I'm not jinxing it right now 😕

Same here...all good. Was traveling last week and was glad to see everything was working properly. Hope it's resolved for good now.

catsmanac commented 2 days ago

Did you get like a Envoy firmware update? Or magically solved at the Enphase side.