Enphase API Security Update breaks integration

[chase314]

The problem

My Enphase Envoy integration stopped working yesterday evening (2/28). While diagnosing the issue, I noticed that Emphase had sent an email titled "Security enhancements to Enphase IQ Gateway API" announcing some security updates being rolled out. I've included most of the email I received below for reference, but they did include a link to updated API Documentation Here.

It appears that these changes are being applied automatically and are breaking the existing API integration for Home Assistant.

What version of Home Assistant Core has the issue?

Home Assistant 2023.2.3

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Enphase Envoy

Link to integration documentation on our website

https://www.home-assistant.io/integrations/enphase_envoy

Diagnostics information

home-assistant_enphase_envoy_2023-03-01T17-27-15.663Z.log

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2023-03-01 08:05:18.744 WARNING (MainThread) [homeassistant.config_entries] Config entry 'Envoy 202202141489' for enphase_envoy integration not ready yet: Error communicating with API: All connection attempts failed; Retrying in background

2023-03-01 08:15:24.975 ERROR (MainThread) [homeassistant.components.enphase_envoy] Unexpected error fetching envoy Envoy 202202141489 data: Could not connect or determine Envoy model. Check that the device is up at 'http://192.168.211.16'.

Additional information

The email included: At Enphase, we take security seriously. We want to ensure that all customers and stakeholders have access to the most secure and reliable operating environment possible.

We’ll be updating the API security protocols associated with the software running on the IQ Gateway, and we’re writing to share information about these changes with all Enphase homeowners, installers, software developers, and partners who may be affected.

These updates have begun propagating across accounts and will continue to roll out over time to all accounts. If you’re creating, using, or maintaining custom monitoring software that relies on interactions with IQ Gateway local interfaces, formally known as Envoy, this critical information will require your review and potential action.

Here is a summary of the changes that will go into effect with release 07.03.120 and higher: • Added a new capability to generate and authenticate secure access tokens via web UI to secure all custom applications and API calls. • Documentation now includes examples of how to use URLs to get tokens programmatically using shell script-based or Python-based methods. • Revised documentation also explains how to connect securely using the updated IQ Gateway local UI and/or IQ Gateway APIs.

[home-assistant[bot]]

Hey there @gtdiehl, mind taking a look at this issue as it has been labeled with an integration (enphase_envoy) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `enphase_envoy` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign enphase_envoy` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

enphase_envoy documentation enphase_envoy source _{^{(message by IssueLinks)}}

[djansen1987]

I have recently got my system as well with the Envoy-S-Standard-EU, firmware D7.0.88 and also not able to intergrate the last 6 digits of my serial number. So i am facing the same issue and unable to link my gateway. Hope there is a solution to this

[apt-itude]

I just turned on a new system (IQ Gateway, firmware D7.3.463), and was really excited to integrate with Home Assistant, only to discover this same issue. The docs that OP linked were extremely helpful, although it was confusing to actually generate an API token, but this support thread helped. Once past that, I can hit the API successfully with curl.

[apt-itude]

I found this custom HACS repository which supports the new auth method, and it works for me.

[chase314]

I just turned on a new system (IQ Gateway, firmware D7.3.463), and was really excited to integrate with Home Assistant, only to discover this same issue. The docs that OP linked were extremely helpful, although it was confusing to actually generate an API token, but this support thread helped. Once past that, I can hit the API successfully with curl.

Glad to hear you got it to work! I see you also commented about using an HACS integration, did both methods work? Or was there still issues after you generated an API token? I'll have to try this when I get home.

Sorry for accidentally marking this as closed, pretty new to GitHub

[apt-itude]

I did use HACS to add the custom component, and that custom component actually uses an enphase auth API to get the token for you using your enlighten username and password (you just have to check the "Use Enlighten" box during config). So the whole manual process of requesting an API token is unnecessary with the custom component.

To be clear about my first comment, I was only mentioning the process of generating an API token to add some additional info to the documentation you provided. I was not able to use the API token with the built-in Enphase Envoy integration, just with curl for debugging. Only the custom HACS component supports the new authentication method as far as I know.

[chase314]

I did use HACS to add the custom component, and that custom component actually uses an enphase auth API to get the token for you using your enlighten username and password (you just have to check the "Use Enlighten" box during config). So the whole manual process of requesting an API token is unnecessary with the custom component.

To be clear about my first comment, I was only mentioning the process of generating an API token to add some additional info to the documentation you provided. I was not able to use the API token with the built-in Enphase Envoy integration, just with curl for debugging. Only the custom HACS component supports the new authentication method as far as I know.

Hi there- thank you for the clarification! When you installed the custom integration from HACS, did you remove the originally installed Envoy integration first? I keep getting the error "The Device is Already Configured" when trying update my credentials for the integration after installing the custom one from HACS. Sorry to bug you about it, just wondering what steps you took since it's working for you.

[apt-itude]

I was setting this up from scratch, so I didn't have a previously configured device to worry about. I'm sure you could delete your configuration for the built-in Envoy integration first, but I don't know if that means you'll lose historical data.

[chase314]

I was setting this up from scratch, so I didn't have a previously configured device to worry about. I'm sure you could delete your configuration for the built-in Envoy integration first, but I don't know if that means you'll lose historical data.

Ah that makes sense! I guess it wouldn't be the end of the world if I lost my historical data (it's only been in place since December). Appreciate the extra info!

[Daniel15]

Possible duplicate of #79382

[kanthamohan]

The envoy-dev custom integration uses the enlighten api to get the token, this does not seem to work when the internet is down. I had an internet outage today and the envoy sensors were down for hours until internet was restored. The core integration needs to support local access without the need to call an enlighten api over internet. The enphase documentation says this is possible by generating a token once

[chase314]

The envoy-dev custom integration uses the enlighten api to get the token, this does not seem to work when the internet is down. I had an internet outage today and the envoy sensors were down for hours until internet was restored. The core integration needs to support local access without the need to call an enlighten api over internet. The enphase documentation says this is possible by generating a token once

Honestly that is something I was afraid of when I was looking at the envoy-DEV integration. I have held off on installing the DEV install because I don't want to wipe out my current integration, hoping they figure out a way to fix the original integration soon.

[chase314]

Possible duplicate of #79382

You're right, this appears to be the same issue. Thanks for the heads up!

[perryvvliet]

The envoy-dev custom integration uses the enlighten api to get the token, this does not seem to work when the internet is down. I had an internet outage today and the envoy sensors were down for hours until internet was restored. The core integration needs to support local access without the need to call an enlighten api over internet. The enphase documentation says this is possible by generating a token once

I was able to generate an access token for the Enphase IQ Gateway local interface API via https://entrez.enphaseenergy.com/entrez_tokens

[giveyouup]

any update on fix for this? I tried with HACs custom integration and still no luck

[MallocArray]

Also waiting on an update to the official integration, as I really don't want to have to setup HACS just for this

[WiebKastanje]

Also waiting on an update to the official integration, as I really don't want to have to setup HACS just for this

Me too

[djansen1987]

M2

[chase314]

Just an update from my end, I ended up using the HACS developer integration and that has been working for me since April. I do see that there is activity going on over here, so maybe the issue will eventually be "fixed" for good.

[liquidbrains]

The concerning thing is the person identified below has not been active on github for more than a year.

Hey there @gtdiehl, mind taking a look at this issue as it has been labeled with an integration (enphase_envoy) you are listed as a code owner for? Thanks!

Code owner commands (message by CodeOwnersMention)

enphase_envoy documentation enphase_envoy source (message by IssueLinks)

[djansen1987]

The concerning thing is the person identified below has not been active on github for more than a year.

Hey there @gtdiehl, mind taking a look at this issue as it has been labeled with an integration (enphase_envoy) you are listed as a code owner for? Thanks!

Code owner commands (message by CodeOwnersMention)

enphase_envoy documentation enphase_envoy source (message by IssueLinks)

There is an email address listed on the github page, we could send him an email?

[mbeijen]

Hi there, I just got an Enphase system installed last week and modified Home Assistant to get it working. I can also take over and contribute my changes upstream?

I am a software developer but so far have not contributed to Home Assistant yet. I've just dropped Greg a mail asking if he would be able or willing to review contributions from my side.

[WiebKastanje]

Hi mbeijnen, You are very welcome to take care of neccessary improvements to get latest Enphase system working. I love to have my Enphase system connected to Home Assistant. I'm not a software developer, so I'm depending on the official Home Assistant integrations. If needed, I can test.

[djansen1987]

The concerning thing is the person identified below has not been active on github for more than a year.

Hey there @gtdiehl, mind taking a look at this issue as it has been labeled with an integration (enphase_envoy) you are listed as a code owner for? Thanks!

Code owner commands (message by CodeOwnersMention)

enphase_envoy documentation enphase_envoy source (message by IssueLinks)

There is an email address listed on the github page, we could send him an email?

He does not reply to email aswell, @frenck how to deal with this? The owner of the code does not seem to response to mail or mentions on github. This issue with the integration forces people with an up-to-date Enphase gateway to move to the HACS solution because the new authentication flow is not support and therefore the code is outdated. Can jou help out?

[frenck]

He does not reply to email aswell, @frenck how to deal with this? The owner of the code does not seem to response to mail or mentions on github. This issue with the integration forces people with an up-to-date Enphase gateway to move to the HACS solution because the new authentication flow is not support and therefore the code is outdated. Can jou help out?

This is an open source project, work is not limited to be done by certain people, anyone can contribute. Feel free to jump in and fix/improve the situation.

../Frenck

[jupsoleil]

This is a duplicate of #79382 - better check there.

[jmoutte]

Hi there, I just got an Enphase system installed last week and modified Home Assistant to get it working. I can also take over and contribute my changes upstream?

I am a software developer but so far have not contributed to Home Assistant yet. I've just dropped Greg a mail asking if he would be able or willing to review contributions from my side.

Can you please create a pull request and reference this issue number?

Also, for everyone, the token created on their online token service seem to be valid for one year if requested by the "system owner". I suppose the firmware update has provided some public key that the Envoy device has locally to validate the tokens. So, even if the connection to Internet is lost, the token should allow to call the APIs on the LAN for quite some time.

[Steve2017]

Sadly my Envoy's firmware was updated this morning, so I have lost all my connection between the Envoy and the integration as well.

I wish @mbeijen all the luck with his attempts to update the official integration. I'd be interested to know what modification he made to his installation to get the token access to the API working. Is it a workaround until the integration is updated?

[Daniel15]

Is it a workaround until the integration is updated?

@Steve2017 - Install HACS, then add https://github.com/briancmpbll/home_assistant_custom_envoy as a custom repo. Still works great with the latest Envoy firmware.

Hi there, I just got an Enphase system installed last week and modified Home Assistant to get it working

@mbeijen There's already a working fork of the code at https://github.com/briancmpbll/home_assistant_custom_envoy

[Steve2017]

Is it a workaround until the integration is updated?

Install HACS, then add https://github.com/briancmpbll/home_assistant_custom_envoy as a custom repo. Still works great with the latest Envoy firmware.

Thanks - I have seen that. I assume it is an either/or? The HACS version won't work alongside the (non-working) official integration? I would hope not to lose all my data, but that seems a moot point given I am no longer getting any new data.

Given the update to the official integration could take a while, I might have no choice but to go the HACS route.

[Daniel15]

@Steve2017 Data should all be preserved fine if you ensure the entity IDs remain the same. Data in Home Assistant just cares about the entity ID; it doesn't care about exactly which integration provides the entity.

[Steve2017]

@Daniel15 I just took the plunge and installed that HACS version. It worked flawlessly (well apart from having to rename 21 inverters to maintain data history)

Hopefully that can be folded into the official version.

I notice it has a new binary sensor - on grid connection. Mine is showing not connected, even though clearly I am connected.

Edit: It seems I spoke too soon. Whenever my Envoy drops connection with Enlighten, it disconnects from HA. Reloading the integration brings the data back in for about a minute. If the Envoy is connected to Enlighten, all is happy in its world.

[fludo]

Same problem, gateway version is: D7.6.175 (f79c8d) Software Build Date 22 Jun, 2023 12:43 PM

Any plan to release a fix for it ?

[mbeijen]

Sadly my Envoy's firmware was updated this morning, so I have lost all my connection between the Envoy and the integration as well.

I wish @mbeijen all the luck with his attempts to update the official integration. I'd be interested to know what modification he made to his installation to get the token access to the API working. Is it a workaround until the integration is updated?

I more or less took the HACS version and tried to 'massage' it into Home Assistant core 'proper'. It works for me, however they use a few non-standard ways of doing stuff -- they use the BeautifulSoup library which is not used in other integrations except for the 'scrape' integration. So I'd want to port this over to more 'normal' ways of Home Assistant programming before submitting the work. However due to 'summer' I'll have no possibility to work on this in the upcoming 4 weeks. If anyone else will want to have a go in the mean time, please do!

[sophof]

Sadly my Envoy's firmware was updated this morning, so I have lost all my connection between the Envoy and the integration as well. I wish @mbeijen all the luck with his attempts to update the official integration. I'd be interested to know what modification he made to his installation to get the token access to the API working. Is it a workaround until the integration is updated?

I more or less took the HACS version and tried to 'massage' it into Home Assistant core 'proper'. It works for me, however they use a few non-standard ways of doing stuff -- they use the BeautifulSoup library which is not used in other integrations except for the 'scrape' integration. So I'd want to port this over to more 'normal' ways of Home Assistant programming before submitting the work. However due to 'summer' I'll have no possibility to work on this in the upcoming 4 weeks. If anyone else will want to have a go in the mean time, please do!

Can you open up your code for what you've done so far? Currently the biggest issue here appears to be that people are a bit confused about the open source method. You can check how to set up a development environment here: https://developers.home-assistant.io/docs/development_environment

Just share your fork on github and link it here, then someone else can follow up on your work if they feel like it in the coming 4 weeks. As far as I can see there's now 4 different people working on a solution all on their own, the trick is to combine these efforts.

No promises, but I might have some time in about a week ;)

[mbeijen]

@sophof my fork is here -- it still fails some pre-commit hooks from home-assistant

https://github.com/mbeijen/home-assistant/tree/enphase-envoy-v7-firmware-support

[nils-82]

Hi everyone,

I'm encountering the same problem after an unwanted enphase firmware update.

I'm using the latest home assistant version 2023.7.1. Hope this can be sorted somehow.

Have a nice day

[catsmanac]

@sophof my fork is here -- it still fails some pre-commit hooks from home-assistant

https://github.com/mbeijen/home-assistant/tree/enphase-envoy-v7-firmware-support

@mbeijen forked yours to see how to help. No experience with adding to ha core, just with the new envoy fw. Or should I fork from @sophof?

[catsmanac]

Sadly my Envoy's firmware was updated this morning, so I have lost all my connection between the Envoy and the integration as well. I wish @mbeijen all the luck with his attempts to update the official integration. I'd be interested to know what modification he made to his installation to get the token access to the API working. Is it a workaround until the integration is updated?

I more or less took the HACS version and tried to 'massage' it into Home Assistant core 'proper'. It works for me, however they use a few non-standard ways of doing stuff -- they use the BeautifulSoup library which is not used in other integrations except for the 'scrape' integration. So I'd want to port this over to more 'normal' ways of Home Assistant programming before submitting the work. However due to 'summer' I'll have no possibility to work on this in the upcoming 4 weeks. If anyone else will want to have a go in the mean time, please do!

Can you open up your code for what you've done so far? Currently the biggest issue here appears to be that people are a bit confused about the open source method. You can check how to set up a development environment here: https://developers.home-assistant.io/docs/development_environment

Just share your fork on github and link it here, then someone else can follow up on your work if they feel like it in the coming 4 weeks. As far as I can see there's now 4 different people working on a solution all on their own, the trick is to combine these efforts.

No promises, but I might have some time in about a week ;)

As this now seems to have become the coordination thread for adding V7 support to enphase envoy to let you know I'm running ENVOY-S EU standard (non-metered) so only can test that type (still working on test env for HA Core changes). We would need testers for ENVOY-S Metered, ENVOY-S metered with no CT's connected, ENVOY-S 3 phase systems and also for V5 firmware and older envoy types for backward compatibility testing.

I have forked @mbeijen ha core and for now created 3 commits based on his enphase-envoy-v7-firmware-support branch

[sophof]

@sophof my fork is here -- it still fails some pre-commit hooks from home-assistant https://github.com/mbeijen/home-assistant/tree/enphase-envoy-v7-firmware-support

@mbeijen forked yours to see how to help. No experience with adding to ha core, just with the new envoy fw. Or should I fork from @sophof?

My fork doesn't contain the branch (yet). an easy way to contribute would be to fork from @mbeijen and do a pull request once you have something you think will contribute. This way the work will stay concentrated in one spot. Today is my last day at work before I have a long vacation, so I'm hoping to devote some time to it as well next week if still necessary. I haven't looked at the code yet, but I get the sense that this is mostly a solved issue, we 'just' need to consolidate.

[edit] just saw @catsmanac contribution, exactly like that :)

[wschoot]

As this now seems to have become the coordination thread for adding V7 support to enphase envoy to let you know I'm running ENVOY-S EU standard (non-metered) so only can test that type (still working on test env for HA Core changes). We would need testers for ENVOY-S Metered, ENVOY-S metered with no CT's connected, ENVOY-S 3 phase systems and also for V5 firmware and older envoy types for backward compatibility testing.

3 phase here and eager to start testing!

[catsmanac]

FYI: Got my HA Core dev environment going. Now working on removing BS4 / BeautifulSoup as that is in the no-go zone of HA Core and fails checks. 3phase was added by @posixx to the @briancmpbll custom integration but still needs to be added here.

[catsmanac]

FYI, The BeautifulSoup has been replaced and @posixx 3 phase support is now migrated as well, but untested as I have no 3phase. Now working on some more recent changes to migrate.

[Daniel15]

Nice work @catsmanac

[catsmanac]

FYI, added the configuration option for scan interval time that was recently added to @briancmpbll. And passed the hurdle of HA pre-commit tests, though not without changes, 'Interesting experience' I'd say. There's some more recent adds worthwhile migrating, but current PR is probably test worthy. I tested it with plain vanilla Envoy-s standard, nothing else.

[catsmanac]

For those that want to test I've packaged the new code as a custom integration that can be installed with HACS. As it is sourced from @briancmpbll repository that is currently actively maintained by @posixx, this test version is slightly behind that one and differs from it by all changes needed to let it pass all the HA Core code Linting and pre-commit tests. There's also some other features added from @vincentwolsink repository.

[chrismfield]

Thank you - that has seems to have worked for me.

[michelcve]

Great work @catsmanac! Do you expect this to land in stable with 2023.8 ?

[catsmanac]

That will probably depend on having testers for the old Envoy types. As long as this is a custom integration that is less of an issue, but if the core one gets replaced and it breaks the old Envoy model that would be a bigger issue as there's no custom integration for that afaik. Testing with new models is needed as well, but the current custom integrations are fallback if something fails.

Currently I'm working in the @briancmpbll custom repo on issues that were reported with old envoy models and then bring them into this core dev work.

Short answer, it's probably tight.

[michelcve]

Right, I understand. Thanks for the hard work @catsmanac, it's really appreciated!