search

jesserizzo / envoy_reader

MIT License
37 stars 26 forks source link

Enphase integration not working after Envoy updated to D7.0.43 (8ed8ea) #78

Open mikeykau opened 3 years ago

mikeykau commented 3 years ago

Authentication process appears to have changed.

peterwalkowiak commented 2 years ago

Peterwalkowiak, Is there some documentation that shows this integration is now merged with HA Core? Can't seem to find anything. …

Something happened with my HA install and the custom showed up as the core for me. I deleted my comment because I realized I was wrong shortly after posting. Still crossing my fingers!

anthony4000 commented 2 years ago

I have two Envoys. 1 running version 5.05 and another with version 7.3. This version (HACs) works great with v7.3. However, now I'm unable to use it with v5.05. Integration just says retrying setup. I'm using IP, (user)envoy, serial. If I remove HACs component and revert to core, v5.05 works with no issues.

Log:

Logger: custom_components.enphase_envoy Source: custom_components/enphase_envoy/envoy_reader.py:721 Integration: Enphase Envoy First occurred: 7:24:04 PM (11 occurrences) Last logged: 7:31:48 PM Unexpected error fetching envoy Envoy 202140124443 data: 0 Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 190, in _async_refresh self.data = await self._async_update_data() File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 150, in _async_update_data return await self.update_method() File "/config/custom_components/enphase_envoy/init.py", line 63, in async_update_data battery_data = await envoy_reader.battery_storage() File "/config/custom_components/enphase_envoy/envoy_reader.py", line 721, in battery_storage if len(ensemble_json) > 0 and "devices" in ensemble_json[0].keys(): KeyError: 0

djk121 commented 2 years ago

Is this going to get fixed? There are several PRs that fix the authentication errors reported in this issue, but they've been sitting there for ~months with no comment/response.

enkrypt3d commented 2 years ago

https://github.com/briancmpbll/home_assistant_custom_envoy I ended up using this and it has been working well for me.

On Fri, Apr 29, 2022, 6:03 PM djk121 @.***> wrote:

Is this going to get fixed? There are several PRs that fix the authentication errors reported in this issue, but they've been sitting there for ~months with no comment/response.

— Reply to this email directly, view it on GitHub https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1113791700, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRTTEIVJ5QWPQDSCBKLVHRMEFANCNFSM465KDKNA . You are receiving this because you were mentioned.Message ID: @.***>

djk121 commented 2 years ago

:(

So is this abandoned? Is it worth sending a PR to the HA folks to switch away from this to that?

jesserizzo commented 2 years ago

Hi all, sorry I've been pretty absent from this. Can anyone on firmwares other than 7.0.43 and 5.0.49 test this out to make sure it doesn't break anything?

sergeknystautas commented 2 years ago

Hi! I spent the weekend walking through a different token issuing approach for 7.X firmware. I just had my system installed a week ago and was pretty miffed that nothing was working with Home Assistant.

I took a very different approach than this PR takes. First off, my enphase admin screen does not have a radio buttons for commissioned or uncommissioned, so I'm not sure how this would work. My approach ports the JS implementation provided by Enphase when you hit the https://envoy.local/ URL. Their implementation for getting local JWT tokens works like this...

  1. Locally generate 40-character random string, and SHA256 hash it.
  2. Send user to https://entrez.enphaseenergy.com/login with a callback parameter and the SHA256 hash.
  3. If the user enters successful credentials, it creates a tracking code and sends you back to https://envoy.local/auth/callback
  4. Note to get this auth flow working, the Python implementation does not have to hit https://envoy.local/auth/callback. This is just an HTML page that generates an XHR call for the next step, that Python does need to call.
  5. Call https://envoy.local/auth/get_jwt with your original random string and the tracking code. This returns a JSON of an access token.

I got this flow all working, with some python JWT parsing to check if the token has expired and automatically refresh the token if needed. Note you have to ignore the SSL verification and the warnings from that because the https://envoy.local/ is a self-signed cert. It's fundamentally just 2 HTTPS calls though... 1 to login, and 1 to get_jwt.

I think the benefit of my approach is that you just need to configure your username/password, and this will handle continually refreshing the JWT as needed. Also it involves no cookies or HTML parsing, which I think will make it less likely to break over time. And ultimately then you can use that JWT then for whatever API calls you want.

Note I'm doing this against D7.3.75 (47f7da)

enkrypt3d commented 2 years ago

No need to reinvent the wheel. This has already been fixed in many different integrations.

On Mon, May 2, 2022, 12:06 AM sergeknystautas @.***> wrote:

Hi! I spent the weekend walking through a different token issuing approach for 7.X firmware. I just had my system installed a week ago and was pretty miffed that nothing was working with Home Assistant.

I took a very different approach than this PR takes. First off, my enphase admin screen does not have a radio buttons for commissioned or uncommissioned, so I'm not sure how this would work. My approach ports the JS implementation provided by Enphase when you hit the https://envoy.local/ URL. Their implementation for getting local JWT tokens works like this...

  1. Locally generate 40-character random string, and SHA256 hash it.
  2. Send user to https://entrez.enphaseenergy.com/login with a callback parameter and the SHA256 hash.
  3. If the user enters successful credentials, it creates a tracking code and sends you back to https://envoy.local/auth/callback
  4. Note to get this auth flow working, the Python implementation does not have to hit https://envoy.local/auth/callback. This is just an HTML page that generates an XHR call for the next step, that Python does need to call.
  5. Call https://envoy.local/auth/get_jwt with your original random string and the tracking code. This returns a JSON of an access token.

I got this flow all working, with some python JWT parsing to check if the token has expired and automatically refresh the token if needed. Note you have to ignore the SSL verification and the warnings from that because the https://envoy.local/ is a self-signed cert. It's fundamentally just 2 HTTPS calls though... 1 to login, and 1 to get_jwt.

I think the benefit of my approach is that you just need to configure your username/password, and this will handle continually refreshing the JWT as needed. Also it involves no cookies or HTML parsing, which I think will make it less likely to break over time. And ultimately then you can use that JWT then for whatever API calls you want.

Note I'm doing this against D7.3.75 (47f7da)

— Reply to this email directly, view it on GitHub https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1114472503, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRXLPX3VHVJ7YPDBQEDVH5ICXANCNFSM465KDKNA . You are receiving this because you were mentioned.Message ID: @.***>

sergeknystautas commented 2 years ago

https://github.com/briancmpbll/home_assistant_custom_envoy doesn't work for me. I get the log message Exception: Could not get 6 month token: You need to sign in first. I'm not sure why it's giving an error at that point, but if I manually recreate the URL it's hitting, I get a good token ~that expired 3 days ago. AFAICT, this is ultimately relying on a working admin system to manually create a long-lived token. The implementation I wrote takes no manual steps or reminders.~

Ok, gonna try to debug what's going on and why this isn't able to hit /entrez-auth-token to get the token when I can get it through my browser.

enkrypt3d commented 2 years ago

You need to login with your enlighten account and tick the box to use enlighten. And put in your site ID and serial number.

On Mon, May 2, 2022, 12:28 AM sergeknystautas @.***> wrote:

https://github.com/briancmpbll/home_assistant_custom_envoy doesn't work for me. I get the log message Exception: Could not get 6 month token: You need to sign in first. I'm not sure why it's giving an error at that point, but if I manually recreate the URL it's hitting, I get a token that expired 3 days ago. AFAICT, this is ultimately relying on a working admin system to manually create a long-lived token. The implementation I wrote takes no manual steps or reminders.

— Reply to this email directly, view it on GitHub https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1114486147, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRWVHFNHWE5JE56MPS3VH5KXVANCNFSM465KDKNA . You are receiving this because you were mentioned.Message ID: @.***>

sergeknystautas commented 2 years ago

The popup to configure the component shows the MAC address, asks for my username and password, a field for the serial number, and a checkbox to use enlighten. I don't have a field for site ID, and not sure what that's used for.

sergeknystautas commented 2 years ago

Tailing the logs revealed it, but I sorta gave it away... the component is asking me to configure the MAC address. The logs are successfully getting the token, but then it's trying to make https calls to a MAC address and can't connect, so the auth flow is fine, it's just the way maybe zeroconf is identifying where to connect to Envoy.

enkrypt3d commented 2 years ago

Why would it be trying to use the Mac address and not the ip? You need to specify the local ip address.

On Mon, May 2, 2022, 12:43 AM sergeknystautas @.***> wrote:

Tailing the logs revealed it, but I sorta gave it away... the component is asking me to configure the MAC address. The logs are successfully getting the token, but then it's trying to make https calls to a MAC address and can't connect, so the auth flow is fine, it's just the way maybe zeroconf is identifying where to connect to Envoy.

— Reply to this email directly, view it on GitHub https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1114493164, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRRSPRHIOG7MC5CXWETVH5MQJANCNFSM465KDKNA . You are receiving this because you were mentioned.Message ID: @.***>

sergeknystautas commented 2 years ago

I believe HA is using zeroconf to discover devices and suggest integrations. When I deleted that suggestion and manually adding the integration with the IP address, everything worked. Well, several failures to connect for unknown reasons, but finally it worked so I'm calling it a win.

efibc commented 2 years ago

sergeknystautas,

Do you have your code posted somewhere for your method of obtain the auth jwt token? I am not trying to integrate with home assist, I am just trying to make an api call to store the various json readings for different dashboards and historical data. So would be interested in trying your code to authenticate and how you make the call to pull the json files. I have tried the curl method using the token and that does not seem to work.

thanks

enkrypt3d commented 2 years ago

https://github.com/jesserizzo/envoy_reader/blob/8486ef1faea1b9b30b96972be491b6a9be436805/envoy_reader/envoy_reader.py

This is the get JWT token python script....

On Thu, Jun 2, 2022 at 10:42 PM efibc @.***> wrote:

sergeknystautas,

Do you have your code posted somewhere for your method of obtain the auth jwt token? I am not trying to integrate with home assist, I am just trying to make an api call to store the various json readings for different dashboards and historical data. So would be interested in trying your code to authenticate and how you make the call to pull the json files. I have tried the curl method using the token and that does not seem to work.

thanks

— Reply to this email directly, view it on GitHub https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1145530433, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRRZTCVQ35J4J654SJLVNFWIHANCNFSM465KDKNA . You are receiving this because you were mentioned.Message ID: @.***>

bmcclure commented 2 years ago

Nothing here seems to be working for me. Trying to use this with https://github.com/briancmpbll/home_assistant_custom_envoy, I can manually follow the token flow and get a token myself just fine, but trying to set up the integration through this script ends up with "Exception: Could not get 6 month token: You need to sign in first" every time.

I've tried ignoring the zeroconf suggestion and setting it up manually but it still fails with the exact same error, so my issue might be different than @sergeknystautas although the symptoms are the same.

sergeknystautas commented 2 years ago

Not that it's a great answer, but I simply removed and added back the installation multiple times, clicked around the envoy admin system (not sure if or what I did that changed anything), and then magically it worked. As you say, you have to ignore the zeroconf suggestion to get the right error message.

Since then, whatever updated on Envoy's side, and now I've had it working soon after my earlier comment. There's just a lot of error handling this integration could add to improve first-time user experience.

bmcclure commented 2 years ago

So far I can't reproduce your results. I've clicked around the Envoy admin side, restarted HA several times, and tried to add the integration at least 30 times at this point. The error is 100% consistent so I don't think it's a matter of trying until it works, but I could be wrong.

enkrypt3d commented 2 years ago

I switched mine to this HACS repo and it has been working great ofr me....

On Fri, Jun 24, 2022 at 11:55 AM Ben McClure @.***> wrote:

So far I can't reproduce your results. I've clicked around the Envoy admin side, restarted HA several times, and tried to add the integration at least 30 times at this point. The error is 100% consistent so I don't think it's a matter of trying until it works, but I could be wrong.

— Reply to this email directly, view it on GitHub https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1165708689, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRXJBVIJKR3J6NNYKSLVQXK7JANCNFSM465KDKNA . You are receiving this because you were mentioned.Message ID: @.***>

enkrypt3d commented 2 years ago

https://github.com/briancmpbll/home_assistant_custom_envoy clicked send without the link! Sorry

On Fri, Jun 24, 2022 at 11:59 AM Corey J @.***> wrote:

I switched mine to this HACS repo and it has been working great ofr me....

On Fri, Jun 24, 2022 at 11:55 AM Ben McClure @.***> wrote:

So far I can't reproduce your results. I've clicked around the Envoy admin side, restarted HA several times, and tried to add the integration at least 30 times at this point. The error is 100% consistent so I don't think it's a matter of trying until it works, but I could be wrong.

— Reply to this email directly, view it on GitHub https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1165708689, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRXJBVIJKR3J6NNYKSLVQXK7JANCNFSM465KDKNA . You are receiving this because you were mentioned.Message ID: @.***>

jesserizzo commented 2 years ago

Sorry, I've been so MIA from this project. Is this problem something that would be fixed by pull request #91 ?

sambeetm commented 2 years ago

Keep getting following error. I am using my enlighten account User/Password, check box 'Use Enlighten'. Any suggestion ??


2022-07-02 00:21:38 DEBUG (MainThread) [custom_components.enphase_envoy.envoy_reader] Checking Token value: 
2022-07-02 00:21:38 DEBUG (MainThread) [custom_components.enphase_envoy.envoy_reader] Found empty token: 
2022-07-02 00:21:40 ERROR (MainThread) [custom_components.enphase_envoy.config_flow] Unexpected exception
Traceback (most recent call last):
  File "/config/custom_components/enphase_envoy/config_flow.py", line 154, in async_step_user
    envoy_reader = await validate_input(self.hass, user_input)
  File "/config/custom_components/enphase_envoy/config_flow.py", line 46, in validate_input
    await envoy_reader.getData()
  File "/config/custom_components/enphase_envoy/envoy_reader.py", line 359, in getData
    await self._getEnphaseToken()
  File "/config/custom_components/enphase_envoy/envoy_reader.py", line 252, in _getEnphaseToken
    token_json = await self._fetch_owner_token_json()
  File "/config/custom_components/enphase_envoy/envoy_reader.py", line 240, in _fetch_owner_token_json
    raise Exception("Could not get 6 month token: " + msg)
Exception: Could not get 6 month token: You need to sign in first```
bmcclure commented 2 years ago

That's the same error I get now! For some people if they just keep retrying it apparently eventually works, but not for me.

enkrypt3d commented 2 years ago

Try deleting the integration and re-add

On Sat, Jul 2, 2022, 10:04 AM Ben McClure @.***> wrote:

That's the same error I get now! For some people if they just keep retrying it apparently eventually works, but not for me.

— Reply to this email directly, view it on GitHub https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1172904175, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRTU6I3MPZPVFETFLZTVSBEADANCNFSM465KDKNA . You are receiving this because you were mentioned.Message ID: @.***>

sambeetm commented 2 years ago

A month back after few tries, it had worked. It stopped reporting yesterday. So far I have tried atleast 30 times without any success. Removed the integration and re-downloaded twice. No success..

sergeknystautas commented 2 years ago

To clarify or emphasize the above point, I was both reinstalling the integration and clicking around the Envoy admin system. I would guess if you redownloaded/reinstalled many times, what triggered it working for me was something I clicked in the admin system.

I'm not sure how much patience you have for this, but you're in a great spot with an account that hasn't enabled that feature, so you could click a bunch of things and fingers crossed, it starts to work for you, you can share what were the last actions you did. Also if you are technical, you might be able to debug where the integration is dropping, which would be helpful to let someone create a better error message in this case.

colinleroy commented 1 year ago

Hi @jesserizzo, I took the liberty of asking them to at least provide a token retrieval API instead of forcing you and other open-source developers to screenscrape and then maintain the screenscraper each time they change their frontend. Hope this helps Colin

enkrypt3d commented 1 year ago

Abandoned? It has been updated and working fine for me. I only switched to the custom component as it has battery support. what issue are you having exactly?

See my post above to download this integration......

cd /usr/src/homeassistant/homeassistant/components/enphase_envoy/

This is the path in my docker container. yours may be a little different. be sure to empty that folder out and ensure u have all files:

wget https://raw.githubusercontent.com/gtdiehl/core/envoy_new_fw/homeassistant/components/enphase_envoy/__init__.py wget https://raw.githubusercontent.com/gtdiehl/core/envoy_new_fw/homeassistant/components/enphase_envoy/config_flow.py wget https://raw.githubusercontent.com/gtdiehl/core/envoy_new_fw/homeassistant/components/enphase_envoy/const.py wget https://raw.githubusercontent.com/gtdiehl/core/envoy_new_fw/homeassistant/components/enphase_envoy/manifest.json wget https://raw.githubusercontent.com/gtdiehl/core/envoy_new_fw/homeassistant/components/enphase_envoy/strings.json wget https://raw.githubusercontent.com/gtdiehl/core/envoy_new_fw/homeassistant/components/enphase_envoy/sensor.py

wget https://raw.githubusercontent.com/gtdiehl/core/envoy_new_fw/homeassistant/components/enphase_envoy/translations/en.json

restart HA

On Fri, Apr 29, 2022 at 6:09 PM djk121 @.***> wrote:

:(

So is this abandoned? Is it worth sending a PR to the HA folks to switch away from this to that?

— Reply to this email directly, view it on GitHub https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1113797156, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRSDTSELUTX5KHWRR5TVHRMYVANCNFSM465KDKNA . You are receiving this because you were mentioned.Message ID: @.***>

roens commented 1 year ago

FWIW, it's nice there's a workaround for this problem. But it doesn't help those of us running HA in Docker. I have paid for and await the delivery of a Home Assistant Yellow with PoE. Until I receive that, I'm running HA in Docker on a server.

Abandoned? It has been updated and working fine for me. I only switched to the custom component as it has battery support. what issue are you having exactly? See my post above to download this integration...... cd /usr/src/homeassistant/homeassistant/components/enphase_envoy/ This is the path in my docker container. yours may be a little different. be sure to empty that folder out and ensure u have all files: wget https://raw.githubusercontent.com/gtdiehl/core/envoy_new_fw/homeassistant/components/enphase_envoy/__init__.py wget https://raw.githubusercontent.com/gtdiehl/core/envoy_new_fw/homeassistant/components/enphase_envoy/config_flow.py wget https://raw.githubusercontent.com/gtdiehl/core/envoy_new_fw/homeassistant/components/enphase_envoy/const.py wget https://raw.githubusercontent.com/gtdiehl/core/envoy_new_fw/homeassistant/components/enphase_envoy/manifest.json wget https://raw.githubusercontent.com/gtdiehl/core/envoy_new_fw/homeassistant/components/enphase_envoy/strings.json wget https://raw.githubusercontent.com/gtdiehl/core/envoy_new_fw/homeassistant/components/enphase_envoy/sensor.py wget https://raw.githubusercontent.com/gtdiehl/core/envoy_new_fw/homeassistant/components/enphase_envoy/translations/en.json restart HA … On Fri, Apr 29, 2022 at 6:09 PM djk121 @.> wrote: :( So is this abandoned? Is it worth sending a PR to the HA folks to switch away from this to that? — Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWHLHRSDTSELUTX5KHWRR5TVHRMYVANCNFSM465KDKNA . You are receiving this because you were mentioned.Message ID: @.>

michaelgaultjr commented 1 year ago

FWIW, it's nice there's a workaround for this problem. But it doesn't help those of us running HA in Docker. I have paid for and await the delivery of a Home Assistant Yellow with PoE. Until I receive that, I'm running HA in Docker on a server.

I'm running HA Docker and I've been using this HACS repository and it's been working perfectly, from my understanding it's using or at least based on the integration by gtdiehl

roens commented 1 year ago

FWIW, it's nice there's a workaround for this problem. But it doesn't help those of us running HA in Docker. I have paid for and await the delivery of a Home Assistant Yellow with PoE. Until I receive that, I'm running HA in Docker on a server.

I'm running HA Docker and I've been using this HACS repository and it's been working perfectly, from my understanding it's using or at least based on the integration by gtdiehl

I must be missing something. On hearing it had worked for you, I tried adding the home_assistant_custom_envoy repo as a custom integration repo via HACS. And while it does offer some additional fields when I attempt to "Add integration", it also fails to add with an "Unexpected error".

rct commented 1 year ago

FYI - Received the following email from Enphase on 2023-04-15. The gist is:

No clue as to how soon it will be. I'm considering reaching out to Enphase Support to see if I can defer the update for a while to make sure the Home Assistant integration gets updated.

Subject: Security enhancements to Enphase IQ Gateway API

To our valued partners and customers, --   At Enphase, we take security seriously. We want to ensure that all customers and stakeholders have access to the most secure and reliable operating environment possible.   We’ll be updating the API security protocols associated with the software running on the IQ Gateway, and we’re writing to share information about these changes with all Enphase homeowners, installers, software developers, and partners who may be affected.   These updates have begun propagating across accounts and will continue to roll out over time to all accounts. If you’re creating, using, or maintaining custom monitoring software that relies on interactions with IQ Gateway local interfaces, formally known as Envoy, this critical information will require your review and potential action.   Here is a summary of the changes that will go into effect with release 07.03.120 and higher: Added a new capability to generate and authenticate secure access tokens via web UI to secure all custom applications and API calls. Documentation now includes examples of how to use URLs to get tokens programmatically using shell script-based or Python-based methods. Revised documentation also explains how to connect securely using the updated IQ Gateway local UI and/or IQ Gateway APIs. Note that if your system is a beta system, an IQ8 system, or an IQ Battery-based system, you may already be running version 07.03.120 or higher on your Gateway.   Complete and newly updated API documentation is available here.   If you have further questions or require additional support after consulting the updated API documentation, please reach out for support.   We’re here to help, and the small investment of time and resources required to enhance these security protocols will be well worth it to safeguard the health and uninterrupted function of your Enphase systems.   Thank you,   The Enphase Team To our valued partners and customers, At Enphase, we take security seriously. We want to ensure that all customers and stakeholders have access to the most secure and reliable operating environment possible. We’ll be updating the API security protocols associated with the software running on the IQ Gateway, and we’re writing to share information about these changes with all Enphase homeowners, installers, software developers, and partners who may be affected. These updates have begun propagating across accounts and will continue to roll out over time to all accounts. If you’re creating, using, or maintaining custom monitoring software that relies on interactions with IQ Gateway local interfaces, formally known as Envoy, this critical information will require your review and potential action. Here is a summary of the changes that will go into effect with release 07.03.120 and higher: * Added a new capability to generate and authenticate secure access tokens via web UI to secure all custom applications and API calls. * Documentation now includes examples of how to use URLs to get tokens programmatically using shell script-based or Python-based methods. * Revised documentation also explains how to connect securely using the updated IQ Gateway local UI and/or IQ Gateway APIs. Note that if your system is a beta system, an IQ8 system, or an IQ Battery-based system, you may already be running version 07.03.120 or higher on your Gateway. Complete and newly updated API documentation is available here. If you have further questions or require additional support after consulting the updated API documentation, please[ reach out for support](https://support.enphase.com/s/contact-us). We’re here to help, and the small investment of time and resources required to enhance these security protocols will be well worth it to safeguard the health and uninterrupted function of your Enphase systems. Thank you, The Enphase Team