search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
71.1k stars 29.75k forks source link

Enphase's "AI Optimization" System Profile not identified by Enphase Envoy integration & can no longer switch profiles via HA #123216

Open joergbattermann opened 1 month ago

joergbattermann commented 1 month ago

The problem

The Enphase Envoy HA integration currently seemingly supports three (hard coded?) storage modes, namely "backup", "self_consumption" and "savings", see i.e, for automations the available options:

image

Enphase however has started pushing a new system profile here in California (and from what I understand also in Germany and a very few other states/countries) called 'AI Optimization' which replaces the previous 'Savings' profile entirely:

image

The current HA Enphase Envoy integration however seems to identify the 'AI Optimization' profile incorrectly as 'Self-Consumption', see me having switched to that new profile today (August 5th 2024) at 2:38pm local time (the small indicator in the pop-up says it at the very bottom):

image

.. while the HA integration reports the following.. quasi that our site was supposedly in 'Self Consumption' all afternoon long:

image

.. which wasn't the case.

I know this issue is somewhere in between a feature request (to support that system profile / storage mode) and an issue.. but I'm leaning a bit towards issue as it shouldn't report a mode that the system wasn't in in case an unsupported system profile / storage mode is available and selected in the Enphase ecosystem.

I am not sure if the 'AI Optimization' system profile already appears explicitly/clearly/distinctly in the API the HA integration uses, but if it does, it would obviously be great if that one was supported one way or another (I switch between storage modes / system profiles via HA) but then again, I don't know if that's possible given it's 'beta' moniker.

What version of Home Assistant Core has the issue?

2024.7.4

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

enphase_envoy

Link to integration documentation on our website

https://www.home-assistant.io/integrations/enphase_envoy

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

home-assistant[bot] commented 1 month ago

Hey there @bdraco, @cgarwood, @joostlek, @catsmanac, mind taking a look at this issue as it has been labeled with an integration (enphase_envoy) you are listed as a code owner for? Thanks!

Code owner commands Code owners of `enphase_envoy` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign enphase_envoy` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

(message by CodeOwnersMention)

enphase_envoy documentation enphase_envoy source (message by IssueLinks)

catsmanac commented 1 month ago

Hi @joergbattermann, what's a company without AI these days.... To be able to see what changed we would appreciate some data from your Envoy using the new options. If you are willing to provide this, can you:

catsmanac commented 1 month ago

Just adding some links on this feature:

catsmanac commented 1 month ago

Was there any error in the log files? Depending on what the envoy reports back for the storage mode it might result in an error and as a result not change from prior state.

Currently these states are recognized:

    BACKUP = "backup"
    SELF_CONSUMPTION = "self-consumption"
    SAVINGS = "economy"
    LEGACY_SAVINGS = "savings-mode"

and both savings-mode and economy are shown as economy mode.

cgarwood commented 1 month ago

One thing I've noticed is a few features within the Enlighten app don't translate directly to settings on the Envoy. For example, there's no "Storm Guard" mode on the Envoy itself. Enabling "Storm Guard" in the Enlighten app just tells Enlighten to modify your system profile when watches are issued, so the only thing the Envoy sees (and thus all HA can see) is that the profile was changed to "Full Backup" -- not that it was changed because of a weather alert.

The processing power in the Envoy is rather limited, so I'd bet most of the AI stuff is running on the Enlighten cloud and the Envoy is completely unaware of it and just responding to changes to import/export limits pushed down from the cloud.

joergbattermann commented 1 month ago

Was there any error in the log files? Depending on what the envoy reports back for the storage mode it might result in an error and as a result not change from prior state.

Currently these states are recognized:

    BACKUP = "backup"
    SELF_CONSUMPTION = "self-consumption"
    SAVINGS = "economy"
    LEGACY_SAVINGS = "savings-mode"

and both savings-mode and economy are shown as economy mode.

No errors in HA's logs when I had switched from 'Self Consumption' to 'AI Optimization' and then back a few hours later.. and as mentioned above/originally, the integration thought it was in Self Consumption all day.

I'll cycle through the different modes now and take the diagnostics with the test fixtures. Is there a wait time I should.. well.. wait inbetween switching modes in Enlighten and for it to propagate to the Envoy / HA integration? I'm asking because after switching from Self-Consumption to Full Backup the behavior had changed quite quickly in accordance with the Full Backcup mode (read: charge batteries with 100% what the PV provides and cover house load entirely from the grid) but the diagnostic data still said 'self-consumption' as storage_settings > mode value

catsmanac commented 1 month ago

. Is there a wait time I should.. well.. wait

Really No idea. Maybe at least 5 min as that is the interval the Envoy reads the inverters. And I think 15 min is the update interval Envoy updates the Enphase cloud. So maybe 15 min!?

joergbattermann commented 1 month ago

Ha... well.. I waited over 15 minutes after switching from Self-Consumption to Full Backup mode and this is what the '/admin/lib/tarif*f' looks like (see the storage_settings > mode value):

"/admin/lib/tariff": "{\"tariff\":{\"currency\":{\"code\":\"USD\"},\"logger\":\"mylogger\",\"date\":\"1722602347\",\"storage_settings\":{\"mode\":\"self-consumption\",\"operation_mode_sub_type\":\"prioritize-energy\",\"reserved_soc\":20.0,\"very_low_soc\":10,\"charge_from_grid\":false,\"date\":\"1722551964\",\"opt_schedules\":true},\"single_rate\":{\"rate\":0.36445000767707827,\"sell\":0.0},\"seasons\":[{\"id\":\"summer\",\"start\":\"6/1\",\"days\":[{\"id\":\"weekdays\",\"days\":\"Mon,Tue,Wed,Thu,Fri\",\"must_charge_start\":0,\"must_charge_duration\":0,\"must_charge_mode\":\"CP\",\"peak_rule\":\"DL\",\"enable_discharge_to_grid\":false,\"periods\":[{\"id\":\"filler\",\"start\":0,\"rate\":0.36445000767707827},{\"id\":\"period_2\",\"start\":900,\"rate\":0.42302998900413515},{\"id\":\"period_1\",\"start\":960,\"rate\":0.5878000259399414},{\"id\":\"period_3\",\"start\":1260,\"rate\":0.42302998900413515},{\"id\":\"filler\",\"start\":1439,\"rate\":0.36445000767707827}]},{\"id\":\"weekend\",\"days\":\"Sat,Sun\",\"must_charge_start\":0,\"must_charge_duration\":0,\"must_charge_mode\":\"CP\",\"peak_rule\":\"DL\",\"enable_discharge_to_grid\":false,\"periods\":[{\"id\":\"filler\",\"start\":0,\"rate\":0.36445000767707827},{\"id\":\"period_2\",\"start\":900,\"rate\":0.42302998900413515},{\"id\":\"period_1\",\"start\":960,\"rate\":0.5878000259399414},{\"id\":\"period_3\",\"start\":1260,\"rate\":0.42302998900413515},{\"id\":\"filler\",\"start\":1439,\"rate\":0.36445000767707827}]}],\"tiers\":[]},{\"id\":\"winter\",\"start\":\"10/1\",\"days\":[{\"id\":\"weekdays\",\"days\":\"Mon,Tue,Wed,Thu,Fri\",\"must_charge_start\":0,\"must_charge_duration\":0,\"must_charge_mode\":\"CP\",\"peak_rule\":\"DL\",\"enable_discharge_to_grid\":false,\"periods\":[{\"id\":\"filler\",\"start\":0,\"rate\":0.31376999616622927},{\"id\":\"period_2\",\"start\":900,\"rate\":0.3282800018787384},{\"id\":\"period_1\",\"start\":960,\"rate\":0.3513999879360199},{\"id\":\"period_3\",\"start\":1260,\"rate\":0.3282800018787384},{\"id\":\"filler\",\"start\":1439,\"rate\":0.31376999616622927}]},{\"id\":\"weekend\",\"days\":\"Sat,Sun\",\"must_charge_start\":0,\"must_charge_duration\":0,\"must_charge_mode\":\"CP\",\"peak_rule\":\"DL\",\"enable_discharge_to_grid\":false,\"periods\":[{\"id\":\"filler\",\"start\":0,\"rate\":0.31376999616622927},{\"id\":\"period_2\",\"start\":900,\"rate\":0.3282800018787384},{\"id\":\"period_1\",\"start\":960,\"rate\":0.3513999879360199},{\"id\":\"period_3\",\"start\":1260,\"rate\":0.3282800018787384},{\"id\":\"filler\",\"start\":1439,\"rate\":0.31376999616622927}]}],\"tiers\":[]}],\"seasons_sell\":[]},\"schedule\":{\"filename\":\"\",\"source\":\"\",\"date\":\"2024-08-06 17:56:47 UTC\",\"version\":\"\",\"reserved_soc\":30.0,\"very_low_soc\":10,\"charge_from_grid\":false,\"batt_mode\":\"self-consumption\",\"battery_mode\":\"self-consumption\",\"operation_mode_sub_type\":\"\",\"opt_schedules\":false,\"override\":false,\"override_backup_soc\":30.0,\"override_chg_dischg_rate\":0.0,\"override_tou_mode\":\"StorageTouMode_DEFAULT_TOU_MODE\",\"schedule\":{\"Disable\":[{\"Sun\":[{\"start\":0,\"duration\":1440,\"setting\":\"ID\"}]},{\"Mon\":[{\"start\":0,\"duration\":1440,\"setting\":\"ID\"}]},{\"Tue\":[{\"start\":0,\"duration\":1440,\"setting\":\"ID\"}]},{\"Wed\":[{\"start\":0,\"duration\":1440,\"setting\":\"ID\"}]},{\"Thu\":[{\"start\":0,\"duration\":1440,\"setting\":\"ID\"}]},{\"Fri\":[{\"start\":0,\"duration\":1440,\"setting\":\"ID\"}]},{\"Sat\":[{\"start\":0,\"duration\":1440,\"setting\":\"ID\"}]}]}}}", "/admin/lib/tariff_log": "{\"headers\":{\"server\":\"openresty/1.17.8.1\",\"date\":\"Tue, 06 Aug 2024 17:56:47 GMT\",\"content-length\":\"3059\",\"connection\":\"keep-alive\",\"strict-transport-security\":\"max-age=63072000; includeSubdomains\",\"x-frame-options\":\"DENY\",\"x-content-type-options\":\"nosniff\"},\"code\":200}",

.. and I waited a bit over 1 hour after switching from Full Backup to AI Optimization and a HA integration reload later and it (still?) looks like this:

"/admin/lib/tariff": "{\"tariff\":{\"currency\":{\"code\":\"USD\"},\"logger\":\"mylogger\",\"date\":\"1722602347\",\"storage_settings\":{\"mode\":\"self-consumption\",\"operation_mode_sub_type\":\"prioritize-energy\",\"reserved_soc\":20.0,\"very_low_soc\":10,\"charge_from_grid\":false,\"date\":\"1722551964\",\"opt_schedules\":true},\"single_rate\":{\"rate\":0.36445000767707827,\"sell\":0.0},\"seasons\":[{\"id\":\"summer\",\"start\":\"6/1\",\"days\":[{\"id\":\"weekdays\",\"days\":\"Mon,Tue,Wed,Thu,Fri\",\"must_charge_start\":0,\"must_charge_duration\":0,\"must_charge_mode\":\"CP\",\"peak_rule\":\"DL\",\"enable_discharge_to_grid\":false,\"periods\":[{\"id\":\"filler\",\"start\":0,\"rate\":0.36445000767707827},{\"id\":\"period_2\",\"start\":900,\"rate\":0.42302998900413515},{\"id\":\"period_1\",\"start\":960,\"rate\":0.5878000259399414},{\"id\":\"period_3\",\"start\":1260,\"rate\":0.42302998900413515},{\"id\":\"filler\",\"start\":1439,\"rate\":0.36445000767707827}]},{\"id\":\"weekend\",\"days\":\"Sat,Sun\",\"must_charge_start\":0,\"must_charge_duration\":0,\"must_charge_mode\":\"CP\",\"peak_rule\":\"DL\",\"enable_discharge_to_grid\":false,\"periods\":[{\"id\":\"filler\",\"start\":0,\"rate\":0.36445000767707827},{\"id\":\"period_2\",\"start\":900,\"rate\":0.42302998900413515},{\"id\":\"period_1\",\"start\":960,\"rate\":0.5878000259399414},{\"id\":\"period_3\",\"start\":1260,\"rate\":0.42302998900413515},{\"id\":\"filler\",\"start\":1439,\"rate\":0.36445000767707827}]}],\"tiers\":[]},{\"id\":\"winter\",\"start\":\"10/1\",\"days\":[{\"id\":\"weekdays\",\"days\":\"Mon,Tue,Wed,Thu,Fri\",\"must_charge_start\":0,\"must_charge_duration\":0,\"must_charge_mode\":\"CP\",\"peak_rule\":\"DL\",\"enable_discharge_to_grid\":false,\"periods\":[{\"id\":\"filler\",\"start\":0,\"rate\":0.31376999616622927},{\"id\":\"period_2\",\"start\":900,\"rate\":0.3282800018787384},{\"id\":\"period_1\",\"start\":960,\"rate\":0.3513999879360199},{\"id\":\"period_3\",\"start\":1260,\"rate\":0.3282800018787384},{\"id\":\"filler\",\"start\":1439,\"rate\":0.31376999616622927}]},{\"id\":\"weekend\",\"days\":\"Sat,Sun\",\"must_charge_start\":0,\"must_charge_duration\":0,\"must_charge_mode\":\"CP\",\"peak_rule\":\"DL\",\"enable_discharge_to_grid\":false,\"periods\":[{\"id\":\"filler\",\"start\":0,\"rate\":0.31376999616622927},{\"id\":\"period_2\",\"start\":900,\"rate\":0.3282800018787384},{\"id\":\"period_1\",\"start\":960,\"rate\":0.3513999879360199},{\"id\":\"period_3\",\"start\":1260,\"rate\":0.3282800018787384},{\"id\":\"filler\",\"start\":1439,\"rate\":0.31376999616622927}]}],\"tiers\":[]}],\"seasons_sell\":[]},\"schedule\":{\"filename\":\"\",\"source\":\"\",\"date\":\"2024-08-06 19:05:40 UTC\",\"version\":\"\",\"reserved_soc\":30.0,\"very_low_soc\":10,\"charge_from_grid\":false,\"batt_mode\":\"self-consumption\",\"battery_mode\":\"self-consumption\",\"operation_mode_sub_type\":\"\",\"opt_schedules\":false,\"override\":false,\"override_backup_soc\":30.0,\"override_chg_dischg_rate\":0.0,\"override_tou_mode\":\"StorageTouMode_DEFAULT_TOU_MODE\",\"schedule\":{\"Disable\":[{\"Sun\":[{\"start\":0,\"duration\":1440,\"setting\":\"ID\"}]},{\"Mon\":[{\"start\":0,\"duration\":1440,\"setting\":\"ID\"}]},{\"Tue\":[{\"start\":0,\"duration\":1440,\"setting\":\"ID\"}]},{\"Wed\":[{\"start\":0,\"duration\":1440,\"setting\":\"ID\"}]},{\"Thu\":[{\"start\":0,\"duration\":1440,\"setting\":\"ID\"}]},{\"Fri\":[{\"start\":0,\"duration\":1440,\"setting\":\"ID\"}]},{\"Sat\":[{\"start\":0,\"duration\":1440,\"setting\":\"ID\"}]}]}}}", "/admin/lib/tariff_log": "{\"headers\":{\"server\":\"openresty/1.17.8.1\",\"date\":\"Tue, 06 Aug 2024 19:05:40 GMT\",\"content-length\":\"3059\",\"connection\":\"keep-alive\",\"strict-transport-security\":\"max-age=63072000; includeSubdomains\",\"x-frame-options\":\"DENY\",\"x-content-type-options\":\"nosniff\"},\"code\":200}",

Am I reading it correctly that the Envoy thinks it's in self-consumption no matter what is actually selected?

catsmanac commented 1 month ago

Seems to be like that yes. Looks like we need to dive into all details if we can find anything else that changes with this. Can you upload the 3 diagnostics files so we can look into all raw data?

joergbattermann commented 1 month ago

Sure thing, I did it the same as last time.. uploaded to my OneDrive and just let me know when you have the file please, wouldn't want to have serial numbers or what else might be buried in those json files hanging around.

Thanks!

catsmanac commented 1 month ago

Thanks, I have them downloaded so you can remove again. First thing I'll do is change any serial into something like 123456789012, if any.

joergbattermann commented 1 month ago

Thanks @catsmanac appreciate it! Lmk if you need any more info or data or something else that might help.

joergbattermann commented 1 month ago

Hmm.. one more thing: I just tried switching the system profiles via HA / the integration and nothing happened as per Enlighten... even after minutes. This used to work just fine.. but seemingly no more.

I enabled debug logging just before 2:50pm local time & trying to switch to Full Backup & Enable 'Charge from Grid' for the System Controller & uploaded the file to the same directory / link as above.. if you want to take a look / maybe see something in there as well.

catsmanac commented 1 month ago

Two more Enphase info sites on NEM3: https://newsroom.enphase.com/news-releases/news-release-details/enphase-energy-expands-deployments-nem-30-product-solution

https://enphase.com/installers/nem3

catsmanac commented 1 month ago

nothing happened as per Enlighten...

Local we see changes happening. At 14:50:03 there's a command to use Backup mode with charge from grid false. The reply to it shows that mode and status. At 14:50:24 there's another command to use Backup this time charge from grid true. The reply to that command shows charge from grid True as well. At 14:50:49 another command for Backup mode going back to charge from grid false, which status is reflected in the reply. Then at 14:50:51 a command to use self-consumption mode. The 2 replies right after it show it switched to self-consumption in the tariff page.

Typically Envoy updates the Enphase cloud once per 15 min, so these changes may be included with that report, not sure. Where they reflected in the Enphase app in the past? It might be that with the new 'AI optimization mode' coming down from Enphase. these are now ignored. Maybe you have to set the Enphase side to one of the of the older modes for the using local changes again. Don't know, just thinking out loud.

@cgarwood, there seems to be a new field in storage_settings "opt_schedules":true

--
2024-08-06 14:50:03.172 DEBUG (MainThread) [pyenphase.envoy] Sending POST to https://192.168.7.160/admin/lib/tariff
"storage_settings":{"mode":"backup","operation_mode_sub_type":"prioritize-energy","reserved_soc":20.0,"very_low_soc":10,"charge_from_grid":false,"date":"1722551964"}}}'

2024-08-06 14:50:04.021 DEBUG (MainThread) [pyenphase.envoy] Request reply in 0.8 sec from https://192.168.7.160/admin/lib/tariff status 200
"storage_settings":{"mode":"backup","operation_mode_sub_type":"prioritize-energy","reserved_soc":20.0,"very_low_soc":10,"charge_from_grid":false,"date":"1722551964","opt_schedules":true}

2024-08-06 14:50:04.732 DEBUG (MainThread) [pyenphase.envoy] Requesting https://192.168.7.160/admin/lib/tariff 
2024-08-06 14:50:04.846 DEBUG (MainThread) [pyenphase.envoy] Request reply in 0.1 sec from https://192.168.7.160/admin/lib/tariff status 200
"storage_settings":{"mode":"backup","operation_mode_sub_type":"prioritize-energy","reserved_soc":20.0,"very_low_soc":10,"charge_from_grid":false,"date":"1722551964","opt_schedules":true}

--
2024-08-06 14:50:24.581 DEBUG (MainThread) [pyenphase.envoy] Sending POST to https://192.168.7.160/admin/lib/tariff 
"storage_settings":{"mode":"backup","operation_mode_sub_type":"prioritize-energy","reserved_soc":20.0,"very_low_soc":10,"charge_from_grid":true,"date":"1722551964"}

2024-08-06 14:50:25.595 DEBUG (MainThread) [pyenphase.envoy] Request reply in 1.0 sec from https://192.168.7.160/admin/lib/tariff status 200
"storage_settings":{"mode":"backup","operation_mode_sub_type":"prioritize-energy","reserved_soc":20.0,"very_low_soc":10,"charge_from_grid":true,"date":"1722551964","opt_schedules":true}

2024-08-06 14:50:27.029 DEBUG (MainThread) [pyenphase.envoy] Requesting https://192.168.7.160/admin/lib/tariff
2024-08-06 14:50:27.155 DEBUG (MainThread) [pyenphase.envoy] Request reply in 0.1 sec from https://192.168.7.160/admin/lib/tariff status 200:
"storage_settings":{"mode":"backup","operation_mode_sub_type":"prioritize-energy","reserved_soc":20.0,"very_low_soc":10,"charge_from_grid":true,"date":"1722551964","opt_schedules":true}

--
2024-08-06 14:50:49.063 DEBUG (MainThread) [pyenphase.envoy] Sending POST to https://192.168.7.160/admin/lib/tariff 
"storage_settings":{"mode":"backup","operation_mode_sub_type":"prioritize-energy","reserved_soc":20.0,"very_low_soc":10,"charge_from_grid":false,"date":"1722551964"}

2024-08-06 14:50:50.346 DEBUG (MainThread) [pyenphase.envoy] Request reply in 1.3 sec from https://192.168.7.160/admin/lib/tariff status 200
"storage_settings":{"mode":"backup","operation_mode_sub_type":"prioritize-energy","reserved_soc":20.0,"very_low_soc":10,"charge_from_grid":false,"date":"1722551964","opt_schedules":true}

--
2024-08-06 14:50:51.082 DEBUG (MainThread) [pyenphase.envoy] Sending POST to https://192.168.7.160/admin/lib/tariff
"storage_settings":{"mode":"self-consumption","operation_mode_sub_type":"prioritize-energy","reserved_soc":20.0,"very_low_soc":10,"charge_from_grid":false,"date":"1722551964"}

2024-08-06 14:50:51.425 DEBUG (MainThread) [pyenphase.envoy] Requesting https://192.168.7.160/admin/lib/tariff

2024-08-06 14:50:52.168 DEBUG (MainThread) [pyenphase.envoy] Request reply in 1.1 sec from https://192.168.7.160/admin/lib/tariff status 200
"storage_settings":{"mode":"self-consumption","operation_mode_sub_type":"prioritize-energy","reserved_soc":20.0,"very_low_soc":10,"charge_from_grid":false,"date":"1722551964","opt_schedules":true}

2024-08-06 14:50:52.396 DEBUG (MainThread) [pyenphase.envoy] Request reply in 1.0 sec from https://192.168.7.160/admin/lib/tariff status 200
"storage_settings":{"mode":"self-consumption","operation_mode_sub_type":"prioritize-energy","reserved_soc":20.0,"very_low_soc":10,"charge_from_grid":false,"date":"1722551964","opt_schedules":true}
catsmanac commented 1 month ago

As for the diagnostic files:

Still scanning through the other iles.

joergbattermann commented 1 month ago

Thanks so far for the analysis! As a little bit if background info: last winter I used HA to switch from the normally used 'Self Consumption' profile to 'Full Backup' + 'Charge from Grid : true' daily to basically charge the batteries full before the most expensive time of the day would kick in. Solar production in the winter brings the batteries up to maybe 50-60% by around noon-1pm that time of the year and the utility rates start going up at 3pm.. so I basically had an automation set at 1pm, checking if SoC is below a certain threshold and if it was, it would switch to Full Backup+Charge from grid and then back to Self-Consumption once SoC was at 100%.

Those HA triggered profile changes basically changed the systems behaviour within a few seconds.. you could see the pull from the grid starting very, very quickly and way under a minute and Enlighten / the cloud side of Enphase's things also reflected that after maybe 10-15 seconds of the automation running.

That did not happen yesterday .. anymore. The HA triggered Full Backup and Charge from Grid didn't do anything, neither did the system itself pull from the grid nor did Enlighten reflect the supposedly changed-to 'Full Backup' profile.

I'll reproduce that / switch through all three available profiles in a few minutes though with more 'patience' and take screenshots and the debug logs + diagnostics along the way ... and I'll do it from both ends - first going through the system profiles in Enlighten one by one and see if and how that's reflected in HA and then do a second series the other way around, (trying to) change the storage_modes / system profiles via HA and see if and how it's reflected in Enlighten.

joergbattermann commented 1 month ago

Ok, first up.. cycling through profiles in Enlighten and see if and how it's reflected in HA. Starting point is 10:40am (local time), I just rebooted my HA system, enabled debug logs on the Enphase integration and the 'include test fixtures' option.

(Starting) System profile as per Enlighten is 'AI Optimization', see:

image

image

image

and

image

image

image

and

image

image

image

and

image

image

... but let's wait the 5 minutes.

image

joergbattermann commented 1 month ago

Ok, took a little break / had to do something else but next up the series the other way - trying to switch through system profiles / storage_modes from the HA side of things and see if it makes it through to Enlighten / the profile and behavior actually changing.

I rebooted the HA system again, re-enabled debug logging on the Enphase integration and also the inclusion of test fixtures for the diagnostic data.

Starting point is the Enphase system being in the 'AI Optimization' system profile ... as per Enlighten... again - I'll try to see if it makes any difference if the system is in one of the other modes separately a bit later / see in a separate comment below, but anyway.. this is what the system looks before starting:

image

and HA integration:

image

image

image

image

.. HA integration "still" reports being in 'Full backup' with charging from grid 'on' however:

image

image

... but again, nothing changed as per Enlighten:

image

image

and

image

.. and no change in Enlighten again / so far:

image

image

.. and in HA:

image

Next & final run I'll try & perform is switch the system into Self-Consumption mode in Enlighten and same in HA / make sure it reflects that after a reboot and THEN set it to 'Full Backup' in/via HA, wait an hour this time and see if it ever makes it to Enlighten in any shape or form. Will do that in a follow-up comment though and close this one up.

catsmanac commented 1 month ago

That is an impressive well documented set of tests Jörg, oh if all issue were that well reported....

I have downloaded the the 2 zip files from your onedrive and am in for some data crunching. Hours in the day have ended here so that'll be for tomorrow.

First impression is as @cgarwood mentioned already, this now seems to be driven from something different ignoring, the storage_settings in tariff. Considering this is available in Germany and here in the Netherlands as well, it has to be something outside of the IQ System Controllers information. I'll be happy to be proven wrong, but it looks we need to start looking for some new endpoint(s) that at least has these new statuses.

Maybe the new status is visible somewhere on the local envoy webpage when opening it in the browser, you'll need a token to access it. That may give some indication where to search. No warm fuzzy feeling on this one yet, think we're in for quite some searching.

joergbattermann commented 1 month ago

Of course Arie, glad it helps a little!

I also just performed that last step.. switching into one of the 'original' modes (Self-Consumption) make sure both systems (HA & Enlighten) reflect that and then switched to 'Full backup'+'Charge from grid' in HA & waited for over an hour and see if it ever materializes on the system & Enlighten, but alas.. it did not.

I set the system to 'Self-Consumption' in Enlighten at 12:40pm, then I rebooted the HA system once again, checked that it reflected the same logical state in HA & Enlighten at 12:46pm...

image

image

... aaaand then switched in HA to said 'Full backup'+'Charge from grid':

image

image

(which again wasn't reflected in Enlighten)

and then I waited / did something else until just now / ~ 1:55pm and Enlighten still was and is showing the system to be in 'Self-Consumption'. No change after over an hour and as such I'd say.. nope.. whatever is changed/set on the Envoy is ignored either already one way or another by Envoy and/or by Enlighten.

Captured the debug log and diagnostic again and uploaded those as well: image

.. and that's it for now, I guess. Let me know if you need any more data and/or access to our system in Enlighten and/or remotely to our Envoy. I also got installer access to my system if that helps in any shape or form. Just let me know if and what you'd need.

catsmanac commented 1 month ago

For the first up 8/7 10:40am-11:17am test run, cycle on Enphase side 'AI Optimization' -> 'Full Backup' -> 'Self-Consumption' -> 'AI Optimization':

The next up cycling HA side while Enphase side is in the 'AI Optimization' system profile:

The last step cycling HA side while Enphase side is in Self-Consumption:

as such I'd say.. nope.. whatever is changed/set on the Envoy is ignored either already one way or another by Envoy and/or by Enlighten

Certainly seems to be the case. Food for thought and need for internet search I guess.

catsmanac commented 1 month ago

Would that also have changed options to go off-grid? I'm not familiar with how that is done, but if there's some real changes in the FW one wonders.

catsmanac commented 1 month ago

Could Schedules in tariff play a role? There's this new "opt_schedules":true in storage_settings. I saw you have tariff values and schedules.

joergbattermann commented 1 month ago

From my understanding there's at least two sides / places where Enphase configures tariffs / utility rate plans.

  1. (I remember -very- vaguely reading somewhere or hearing my installer say that that's the legacy / no longer used way) In the local Envoy's interface where one can set rates (static or time-of-use ones) AND seemingly also a battery charging (time) schedule. However nothing's configured there for our system/site image image image

  2. In Enlighten and that's what -is- used for the previously Savings- & now Ai Optimization profile and has our utility's tariffs configured. Enlighten does however not have any 'schedules' (again.. as in time scheduling) functionality but they might just use that term to indicate the Time-of-Use rate's time of the day when it switches between the peak and off-peak per-kWh rates: image

catsmanac commented 1 month ago

The numbers in your screenshot differ from what is in tariff as well.

Seems we're running out of options pretty quickly...

joergbattermann commented 1 month ago

Do you know of any way Arie if/how I can put a proxy or something in between the Envoy and the internet / their Enlighten backend to quasi see what the Envoy pulls/gets / sends.. if not the body (due to https etc) but maybe start with what endpoints are being called? At the end of the day the way the system behaves still got to be controlled locally, so I wonder what the system receives once I make a change in the cloud.

I was also just parsing over the official cloud / v4 Enphase API docs and nothing has changed there for 'AI Optimization' specifically for the supported modes, so... something must be different from what Enphase / their cloud sends to the Envoy (or it pulls from the cloud) vs. what the integration sends. I guess..

image

catsmanac commented 1 month ago

Proxy is not something I've played with. Squid seems to be useful in some case, there's probably more out there. It's the Envoy that opens the connection which is used to exchange the information. And it's encrypted I think.

Might be interesting to see how the cloud API shows under these circumstances, but it's very restricted in number of requests one can do, unless you have a paid subscription. As AI Optimization replace the Savings Mode, the status itself is probably not the issue.

There's something else in the Enphase cloud that provide details on time and tariff to the Envoy.

The feature is announced for the Netherlands over here as well using dynamic contracts. That will provide information one day ahead on what energy prices will be by the hour for next day. Over here it's typically midday negative prices caused by a abundance availability of solar power rather then the august-september evening peaks.

And here there's no IQ Combiner boxes, just batteries, pv, Envoy and IQ Relay as grid separator. That may be another drive to do this outside the tariff.

catsmanac commented 1 month ago

Just in search to find some info, the envoy has endpoints /home and /ivp/livedata/status. wondering if these have some added data.

Opening backbone/application.js will return the js code that may have references to new pages with information, mine has a list in b.ApiUrls

joergbattermann commented 1 month ago

application.js.zip

Attached the application.js file Arie (and it's name as per inspector tool is actually referenced with a version parameter "application.js?version=07.00.20") and visually that mentioned section looks like this:

image

Let me know if you see any endpoints that you'd want me to send sample GET or sth requests to

joergbattermann commented 1 month ago

As for /ivp/livedata/status it returns:

{ "info": "Last available status data shown. Live stream not enabled. Enable using /ivp/livedata/stream/ API", "connection": { "mqtt_state": "connected", "prov_state": "configured", "auth_state": "ok", "sc_stream": "disabled", "sc_debug": "disabled" }, "meters": { "last_update": 1723316976, "soc": 26, "main_relay_state": 1, "gen_relay_state": 5, "backup_bat_mode": 1, "backup_soc": 20, "is_split_phase": 1, "phase_count": 2, "enc_agg_soc": 26, "enc_agg_energy": 6600, "acb_agg_soc": 0, "acb_agg_energy": 0, "pv": { "agg_p_mw": 4024321, "agg_s_mva": 4221519, "agg_p_ph_a_mw": 2016139, "agg_p_ph_b_mw": 2008182, "agg_p_ph_c_mw": 0, "agg_s_ph_a_mva": 2111102, "agg_s_ph_b_mva": 2110416, "agg_s_ph_c_mva": 0 }, "storage": { "agg_p_mw": -3484170, "agg_s_mva": -3433765, "agg_p_ph_a_mw": -1742998, "agg_p_ph_b_mw": -1741172, "agg_p_ph_c_mw": 0, "agg_s_ph_a_mva": -1715016, "agg_s_ph_b_mva": -1718749, "agg_s_ph_c_mva": 0 }, "grid": { "agg_p_mw": 32786, "agg_s_mva": 58694, "agg_p_ph_a_mw": 76660, "agg_p_ph_b_mw": -43874, "agg_p_ph_c_mw": 0, "agg_s_ph_a_mva": 1056819, "agg_s_ph_b_mva": -998125, "agg_s_ph_c_mva": 0 }, "load": { "agg_p_mw": 572937, "agg_s_mva": 846448, "agg_p_ph_a_mw": 349801, "agg_p_ph_b_mw": 223136, "agg_p_ph_c_mw": 0, "agg_s_ph_a_mva": 1452905, "agg_s_ph_b_mva": -606458, "agg_s_ph_c_mva": 0 }, "generator": { "agg_p_mw": 0, "agg_s_mva": 0, "agg_p_ph_a_mw": 0, "agg_p_ph_b_mw": 0, "agg_p_ph_c_mw": 0, "agg_s_ph_a_mva": 0, "agg_s_ph_b_mva": 0, "agg_s_ph_c_mva": 0 } }, "tasks": { "task_id": 1357979199, "timestamp": 1723315336 }, "counters": { "main_CfgLoad": 7, "main_CfgChanged": 7, "main_CfgNotFound": 13244, "main_taskUpdate": 215, "MqttClient_publish": 97899, "MqttClient_live_debug": 844, "MqttClient_respond": 1318, "MqttClient_msgarrvd": 681, "MqttClient_reconnect": 6, "MqttClient_create": 8, "MqttClient_setCallbacks": 8, "MqttClient_connect": 8, "MqttClient_subscribe": 8, "SSL_Keys_Create": 8, "sc_SendStreamCtrl": 463, "sc_SendDemandRspCtrl": 1, "rest_Status": 4747, "tunnel_control_msg": 5, "tunnel_control_close": 5, "tunnel_control_open": 5 }, "dry_contacts": { "�\u0011": { "dry_contact_id": "�\u0011", "dry_contact_type": "�\u0011", "dry_contact_load_name": "\u0006", "dry_contact_status": 0 }, "": { "dry_contact_id": "", "dry_contact_type": "", "dry_contact_load_name": "", "dry_contact_status": 0 }, "": { "dry_contact_id": "", "dry_contact_type": "", "dry_contact_load_name": "", "dry_contact_status": 0 }, "": { "dry_contact_id": "", "dry_contact_type": "", "dry_contact_load_name": "", "dry_contact_status": 0 } } }

joergbattermann commented 1 month ago

/home is called regularly by the web interface and returns i.e.:

{ "software_build_epoch": 1722401163, "timezone": "US/Pacific", "current_date": "08/10/2024", "current_time": "12:11", "network": { "web_comm": true, "ever_reported_to_enlighten": true, "last_enlighten_report_time": 1723316332, "primary_interface": "eth0", "interfaces": [ { "signal_strength": 99, "signal_strength_max": 31, "network": false, "type": "cellular", "interface": "ppp0", "dhcp": true, "ip": "10.64.64.64", "carrier": true }, { "type": "ethernet", "interface": "eth0", "mac": "some_other_mac", "dhcp": true, "ip": "192.168.7.160", "signal_strength": 1, "signal_strength_max": 1, "carrier": true }, { "signal_strength": 0, "signal_strength_max": 0, "type": "wifi", "interface": "wlan0", "mac": "some_mac", "dhcp": true, "ip": null, "carrier": false, "supported": true, "present": true, "configured": false, "status": "connecting" } ] }, "tariff": "none", "comm": { "num": 36, "level": 5, "pcu": { "num": 35, "level": 5 }, "acb": { "num": 0, "level": 0 }, "nsrb": { "num": 0, "level": 0 }, "esub": { "num": 1, "level": 5 }, "encharge": [ { "num": 5, "level": 0, "level_24g": 0, "level_subg": 0 } ] }, "wireless_connection": [ { "signal_strength": 0, "signal_strength_max": 0, "type": "BLE", "connected": true } ] }

catsmanac commented 4 weeks ago

Thanks Jörg.

"application.js?version=07.00.20"

That's the same version as on my plain vanilla Envoy with D7.6.175. File compare confirms these are the same code as well. So basically no change in the UI functions it seems.

The livedata status has a backup_bat_mode field which currently has a value of 1 (also see description by @Matthew1471, he doing a impressive and tremendous job on describing the API). Maybe, just maybe this value may reflect the AI Optimization state, but as it's going now I'll doubt it.

    "meters": {
        "last_update": 1723316976,
        "soc": 26,
        "main_relay_state": 1,
        "gen_relay_state": 5,
        "backup_bat_mode": 1,
        "backup_soc": 20,
        "is_split_phase": 1,

The other one to try from the ApiUrls list might be "/ivp/ensemble/profile_status". This one is also in Matthew1471 description with an empty response so might be a doomed attempt as well.

The /home has no real options either as far as I can see from you example.

joergbattermann commented 4 weeks ago

/ivp/ensemble/profile_status returns the following Arie:

[
    {
        "serial_num": "SERIAL_NUMBER_CAN_COMMUNICATION_KIT",
        "device_type": "ENPOWER",
        "admin_state": 2,
        "device_record_type": "AGF",
        "ph_ind": "ph-a",
        "gcpa_ph_ind": "ph-a",
        "man_ph_ind": "ph-unk",
        "status": 2,
        "message": ""
    },
    {
        "serial_num": "SERIAL_NUMBER_SYSTEM_CONTROLLER",
        "device_type": "ENPOWER",
        "admin_state": 2,
        "device_record_type": "AGF",
        "ph_ind": "ph-a",
        "gcpa_ph_ind": "ph-a",
        "man_ph_ind": "ph-unk",
        "status": 2,
        "message": ""
    },
    {
        "serial_num": "SERIAL_NUMBER_BATTERY_1",
        "device_type": "ENCHARGE",
        "admin_state": 2,
        "device_record_type": "AGF",
        "ph_ind": "ph-a",
        "gcpa_ph_ind": "ph-a",
        "man_ph_ind": "ph-unk",
        "status": 2,
        "message": ""
    },
    {
        "serial_num": "SERIAL_NUMBER_BATTERY_2",
        "device_type": "ENCHARGE",
        "admin_state": 2,
        "device_record_type": "AGF",
        "ph_ind": "ph-a",
        "gcpa_ph_ind": "ph-a",
        "man_ph_ind": "ph-unk",
        "status": 2,
        "message": ""
    },
    {
        "serial_num": "SERIAL_NUMBER_BATTERY_3",
        "device_type": "ENCHARGE",
        "admin_state": 2,
        "device_record_type": "AGF",
        "ph_ind": "ph-a",
        "gcpa_ph_ind": "ph-a",
        "man_ph_ind": "ph-unk",
        "status": 2,
        "message": ""
    },
    {
        "serial_num": "SERIAL_NUMBER_BATTERY_4",
        "device_type": "ENCHARGE",
        "admin_state": 2,
        "device_record_type": "AGF",
        "ph_ind": "ph-a",
        "gcpa_ph_ind": "ph-a",
        "man_ph_ind": "ph-unk",
        "status": 2,
        "message": ""
    },
    {
        "serial_num": "SERIAL_NUMBER_BATTERY_5",
        "device_type": "ENCHARGE",
        "admin_state": 2,
        "device_record_type": "AGF",
        "ph_ind": "ph-a",
        "gcpa_ph_ind": "ph-a",
        "man_ph_ind": "ph-unk",
        "status": 2,
        "message": ""
    }
]
catsmanac commented 3 weeks ago

Thanks Jörg, this data doesn't reveal any information either. Running out of options now. Back to square 1 and revisit all the data again to see if I overlooked something.

catsmanac commented 2 weeks ago

Hit this one which seems updated for the AI Optimizations. No real news though

https://support.enphase.com/s/article/how-does-my-enphase-energy-system-with-iq-batteries-work