Envoy-S D7.0.88 firmware

[0crap]

Does this work with the new Envoy-S D7.0.88 firmware?

[H4nsie]

My Envoy-S is on D5.0.62 firmware, which seems to be the latest? (No update available). You can test easily if http://envoy:[last-6-serial]@[ip-of-envoy]/api/v1/production/inverters gives a json response with micro inverter watt production and serial numbers. Please let me know, thanks.

[0crap]

Thx for your quick answer!

Unfortunately not, all that nice stuff has come to an end with the new firmware as I understand. (Security update is what Google tells me why this is done.) I have a brand new system, which came with D7.0.88 pre installed and I would like it to add to my Domoticz. If you switch off and on your Envoy-S you will probably end up with the same issue.

If I type in the exact URL as instructed, it gives a 401 page and gets redirected. As seen in the below screenshot.

[H4nsie]

Someone on Tweakers forum has a solution. https://gathering.tweakers.net/forum/list_message/73148096#73148096

I've requested his modification for adding to my plugin.

Regards, Hans

[0crap]

Great stuff! I played already a bit with my system. If I create a token and paste it in, it comes with a general page that shows the amount of inverters and so on.

Next I tested a few URL's and they do seem to work.

• https://192.168.2.188/api/v1/production/

  
  {
  "wattHoursToday": 7663,
  "wattHoursSevenDays": 4577,
  "wattHoursLifetime": 12241,
  "wattsNow": 341
  }

- https://192.168.2.188/api/v1/production/inverters
Outputs all inverters like for example:
`[
  {
    "serialNumber": "xxxxxxxx",
    "lastReportDate": 1665931886,
    "devType": 1,
    "lastReportWatts": 39,
    "maxReportWatts": 223
  },`

[0crap]

Let me know when your plugin is ready, happy to test. Thx!

[0crap]

Does it help to have a look _here_?

Looks like a Python script, which can work with tokens for the Envoy, used by the Home Assistant boys.

[H4nsie]

Thanks for that script. Also found documentation at Enphase. Wil try later this week to upgrade my Envoy to v7.

[0crap]

Awesome! It should update automagically if you power cycle the Envoy-S. If not you can ask Enphase Support to do it remotely.

Thx for your time!

[H4nsie]

Would you like to test this 1.0.5 version? https://github.com/H4nsie/EnphaseEnvoy/tree/develop Be sure to use the DEVELOPMENT branch. As you told, D5 and D7 firmware use different login methods. If all works well, the plugin determines automatically what to use. Thanks, Hans

[0crap]

Sure! Never done this before, so I have to ask. Can I follow the steps here to the letter? (README.md)

[H4nsie]

Yes, but one change to use de development branche: git clone -b develop https://github.com/H4nsie/EnphaseEnvoy.git So in right order: cd domoticz/plugins git clone -b develop https://github.com/H4nsie/EnphaseEnvoy.git sudo systemctl restart domoticz.service

and then it should appear as 'Enphase Envoy - with micro inverters' in the hardware list to add.

Because I'm still on firmware D5, I'm not sure how to obtain the sessionID, what need to be filled in in the settings. If you are unsure too, please wait till I can update my Enovy. No problem!

[0crap]

Thx! I'm very curious, so I will give it a go and see how far I can get. I'll make a backup of the SD card in my Raspberry Pi first, so no problem if it all fails. Will post my results later this evening.

[H4nsie]

Be aware that obtaining the SessionId can be quite technical. I am not sure if it is the session token from Enohase of it is the resulting cookie when using the token. I have to examine when my envoy is uodated to v7. Verstuurd vanaf mijn iPhoneOp 19 okt. 2022 om 17:57 heeft 0crap @.***> het volgende geschreven: Thx! I'm very curious, so I will give it a go and see how far I can get. I'll make a backup of the SD card in my Rasperry Pi first, so no problem if it all fails. Will post my results later this evening.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[0crap]

Well, I tried a bunch of things, but no sigar. The Error log keeps giving Error: Envoy-S: Plugin not running. Please check parameters and LAN connection and restart plugin At some point the Error log also gave Envoy-S: Error connecting to Enphase Envoy on 192.168.2.188 error: HTTPSConnectionPool(host='192.168.2.188', port=443): Max retries exceeded with url: /info.xml (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1056)'))). Please restart plugin.

I used the sessionID I found, after a login to my Envoy-S using a token, by hitting F12 and used my Chrome browser DevTools.

Might be very well I misunderstood something in the process.

[0crap]

For fun I also tried to add the 'Enphase Envoy - with micro inverters' and pasted the Token itself on the sessionID field. But that gave the same result. :-)

(Small typo on the Domoticz Setup->Hardware tab, that field name has the label sessioID, without the n.)

[H4nsie]

Thanks for trying and the extensive reports. As I can see the fetch of info.xml goes wrong, but I cannot test myself. I requested to update my Envoy to V7 hopefully in a few days.

Thank you very much for your time! I contact you when I have solved this!

Hans

Op 19 okt. 2022, om 20:43 heeft 0crap @.***> het volgende geschreven:

For fun I also tried to add the 'Enphase Envoy - with micro inverters' and pasted the Token itself on the sessionID field. But that gave the same result. :-)

— Reply to this email directly, view it on GitHub https://github.com/H4nsie/EnphaseEnvoy/issues/1#issuecomment-1284428860, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBLOCV4AYZHDAJJL656UALWEA6OJANCNFSM6AAAAAARGJF5Z4. You are receiving this because you commented.

[0crap]

Funny thing is that the info.xml seems to be the only page that does not need a token or whatever. In my browser I do get a message that the site is not secure and I have to click on go anyway to open the page. (Certificate not valid stuff.)

Here you can find the output of that page if you're interested.

[H4nsie]

Ah, so info.xml also is switched to ssl in D7 firmware. You could try adding , verify=false to the command:

systemXML = requests.get('http://' + Parameters["Address"] + '/info.xml', verify=False)

I do a request on info.xml for determining the firmware version, so the plugin is usable for both versions of firmware. In your output is shows D7, in mine D5 (I've requested the installer for firmware update to D7 in the installer portal, but no response yet. )

Dank, Hans

[0crap]

Done that to the plugin.py file. And interesting, it does gives a different error message in Domoticz.

Error: Envoy-S: Error connecting to Enphase Envoy on 192.168.2.188 error: HTTPSConnectionPool(host='192.168.2.188', port=443): Max retries exceeded with url: /production.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1056)')))

[0crap]

Reloaded your dev branch..... So close.

[0crap]

What I still don't get is how this sessionID number even can work in the first place?

I mean, on my Windows PC I open a Chrome browser and enter the IP address of my Envoy-S. Put in my token on the Envoy-S webpage and grab the sessionID from the cookie inside my browser. Next I add the hardware device on my Domoticz, with the sessionID I got from the step above. However my Domoticz runs on a Raspberry Pi, so there is no browser cookie on that device. The sessionID will not point to an existing cookie on the Raspberry Pi, because that cookie is on my Windows PC.

Do I miss the obvious here?

[antoin]

There is some syntax error in the current 1.0.5 test version. I informed H4nsie today. The correct cookie setting should be cookies='sessionid=' + Parameters["SessionID"] (Or try to fill in "sessionid=YourSessionID" in the UI in domoticz. Mind the "sessionid=" prefix that needs to be in the cookie).

Also all the other production URL need the cookie. As you both found out above, all URLs for firmware D7 now also need SSL, where for D5 you only needed credentials for some URLs.

You got the correct cookie from your chrome browser with the SessionID, but it will expire when you close your browser if you don't edit the expire date. If you understand Dutch, I just wrote a manual: https://gathering.tweakers.net/forum/list_message/73214096#73214096

[antoin]

I think it works like this:

• The envoy creates a sessionID for the browser session and sends the cookie to the browser
• Any query to the envoy that presents this cookie will be answered. (Otherwise no proxies would work)
• The cookie will expire when the session closes. In other words, when the browser closes it sends a message to the envoy that the session ended, and the envoy will destroy the sessionID.

So, to make the SessionID cookie to remain working, we must make sure it doesn't expire. You can do this by editing it in the browser and set a long expire date to it, and not only for one session. Now when the browser closes, the sessionID cookie is still valid on the envoy.

[antoin]

Do I miss the obvious here?

You seem to think a cookie is something complicated, but it is just a textfile. In this case the cookie looks like "sessionid='YoursessionID'" with a tag that it's a cookie. So when you copy the "YoursessionID" into domoticz on your PI, Domoticz can create this same cookie too ;-)

[0crap]

There is some syntax error in the current 1.0.5 test version. I informed H4nsie today. The correct cookie setting should be cookies='sessionid=' + Parameters["SessionID"] (Or try to fill in "sessionid=YourSessionID" in the UI in domoticz. Mind the "sessionid=" prefix that needs to be in the cookie).

Also all the other production URL need the cookie. As you both found out above, all URLs for firmware D7 now also need SSL, where for D5 you only needed credentials for some URLs.

You got the correct cookie from your chrome browser with the SessionID, but it will expire when you close your browser if you don't edit the expire date. If you understand Dutch, I just wrote a manual: https://gathering.tweakers.net/forum/list_message/73214096#73214096

Yep Dutch :-) Thanks for the cookie lesson, that I'm able to extract correctly. I changed the script from Hans with the suggestion you gave above. Unfortunately Domoticz keep being a pain in the butt. Can you please have a look at the script here if I made a mistake?

Domoticz error log gives: Error: Envoy-S: Error connecting to Enphase Envoy on 192.168.2.247 error: 'SessionID'

[0crap]

OK, got it working by changing the plugin as seen on the tweakers thread. I added:

• cookies = dict(sessionid='mysessionidfoundinthebrowser')
• on the D7 url's in the script add cookies=cookies to that line

So when the cookie is hard coded into the script, then it all comes to life! The problem seems to be that, when you add the envoy Hardware into Domoticz, the SessionID field is not read correctly into the script. Anyone with a suggestion how to fix this? This last step is above my pay grade. :-)

[H4nsie]

Thank you both foor explaining the cookie. I'm hoping for a firmware update in the near future to continue working on this plugin. As seen in the HA script, automatically retrieving a sessionID is possible and would be nice if we can build this in the plugin, so it auto-renews when the cookie is expired.

On my request to Enphase for updating my Envoy-S from D5 to D7 a received the (in Dutch!) answer this morning:

Goedemorgen,
Momenteel is dat niet mogelijk, aan de hand van serienummers en batches worden de Envoys geupdate.

So unfortunately I have to reboot my Envoy every morning and check the info.xml if it has updated...

[0crap]

Crazy stuff! Bit silly they can't push your envoy on request, must be a simple task for them. Anyway, nothing you can do about it.

As a FYI, you can find a Tech Brief on the D7 update here. On page 3, step 5 it states: "Copy and paste the token in your home automation setup for authentication." I guess that is the best way to go for to make the new script.

Upcoming weekend I have asked a friend to come over and have a look. He has good Python skills. We will see how it goes. If anything comes from it I'll give come feedback here.

[claskfosmic]

Hi all,

Came here via the Tweakers.net forum and think i figured it out! Thanks to the post at https://gathering.tweakers.net/forum/list_message/73135624#73135624 and the Tech Brief on the D7 update on [https://store-d9.enphase.com/download/iq-gateway-access-using-token-tech-brief] (https://store-d9.enphase.com/download/iq-gateway-access-using-token-tech-brief).

Pasting the sessionId in the cookies and executing the URL's on my local envoy (IQ Gateway) it worked. Fetching the token from enlighten.enphaseenergy.com and entrez.enphaseenergy.com also worked. But for some reason, the token didn't work on the URL's on the local envoy.

After a frustrating evening and a lot of debugging, I figured out that the token should be sent to https://{Gateway_ip}/auth/check_jwt. When the token is correct, the IQ Gateway will answer with a page, showing the HTML <!DOCTYPE html><h2>Valid token.</h2> and a cookie with the sessionId. When using this sessionId on the URL's on my IQ Gateway, running on firmware version D7.0.88, it works!

So... In the correct order:

1. Get the info from https://{Gateway_ip}/info.xml, which includes the firmware/software version and the serial number.
2. Use the username and password from your Enlighten account to get the (first) sessionId by executing a POST on https://enlighten.enphaseenergy.com/login/login.json.
3. Use the serial number, username and the sessionId to get the authToken by executing a POST on https://entrez.enphaseenergy.com/tokens.
4. Use the authToken to get the cookies for the IQ Gateway by executing a GET on https://{Gateway_ip}/auth/check_jwt.
5. Use the sessionId from the cookie for the other URL's (like https://{Gateway_ip}/production.json and https://Gateway_ip}/api/v1/production/inverters/ on the IQ Gateway.

Using this steps, it's no longer needed to manually get and insert the sessionId. I can just enter my username and password from the Enlighten account into Domoticz and the plugin will login and handle everything for me.

Didn't tested yet how to handle things like expired tokens, automatic re-login etc, but it's a start.

@H4nsie Sorry... I just saw you're already busy with version 1.0.5, which also includes some adjustments for the D7. software, so some thing may overlap each other... But the "auto login" part might be a nice one to add. What is the best way to sent you my part of the code?

[0crap]

@claskfosmic that is great work done! If you are familiar with GitHub you can open a PR (Pull Request) against the development branch of H4nsie. As you have noticed, the development branch already contains some adjustments for the new D7 firmware.

If that seems a bit daunting, just use Pastebin and put a link in the comments to share your code!

From the Tech Brief you've read as well, I get that it's not the best way to use your Enlighten username and password in the Domoticz script. Just get a token manually, which is valid for one year, and use that token in the Domoticz script to get it all working.

Thx for your time.

[claskfosmic]

Hi, thnx! I tried to create a pull request in the development branch, but I keep getting errors (403 / Unauthorized / Permission Denied), so I placed it on https://pastebin.com/H4TPAajw. I used v1.0.5 from the development branch, added my adjustments and called it v1.0.6.

The sessionId is removed from the Domoticz parameters and replaced by the username and password. When no sessionId is available, the plugin will first login to Enphase, get the authToken, validate it on the local IQ Gateway. On success, the sessionId is stored. (currently in self.sessionId, better would be to store it in Domoticz / plugin options, don't know if this is possible?).

For now, I authorize once everytime Domoticz/the plugin starts, a new sessionId will be fetched. So after an 100% uptime of 1 year, the sessionId needs to be renewed. When the sessionId is stored somewhere in Domoticz/the plugin, it needs to check it and, when expired, generate a new one.

I personally think it's better to use the username and password and let the plugin handle the rest, instead of me having to figure out again, 1 year from now, how I get a new/valid sessionId again in order to get the plugin to work again in Domoticz.

[H4nsie]

WoW, surprise after returning from short holiday break! Great work both of you! I pushed the Pastebin into the development branch with some slight changes (made the username/password not required, so it is usable for D5). Thanks for updating to compatible with D7! I have 1.0.6 now running in my lab-environment (docker with Domoticz container) and I can confirm it working on D5 firmware! If you can confirm it's error-free on D7 I'll merge it to the main branch.

Hopefully my Envoy is updated soon to D7!

Thanks for the good work, Hans

[0crap]

Just tested by pulling in the dev branch with git clone -b develop https://github.com/H4nsie/EnphaseEnvoy.git Works great!

If the hardware debug option is on, you also see the token is fetched correctly. Awesome! The debug does however also show the username and password in clear text. This is also the case when I go to Setup -> Hardware and click on the Envoy entry.

My preference would be feeding in just the Token, on the Domoticz Setup -> Hardware page. Just like the Tech Brief suggests. But hey, not complaining here, happy that it works!

The question now is what happens after one month? For sure the Token is valid for one year, but the derived cookie, at least in my browser, is valid for one month. Guess we just wait and see. :-)

Again, great job done gents!

Edit: Just because I wanted to know, I gave the Envoy-S a power cycle to see what happened. Of course the session is broken so the logging stops. The only thing I needed to do, to get it back alive, was in Domoticz, go to Settings -> Hardware, click on the Envoy-S entry and click "Update". That's all!

[H4nsie]

Thanks, made some changes and pushed to 1.0.6 dev:

• Added password="true" to the parameter field, so the password is hidden in the settings.
• Cannot disable showing password in Debug. Standard al settings fields are logged in debug mode. (Should be the same with other Domoticz plugins)
• When I'm on D7 I'll take a look at username/password vs token. Or make both options available :-) and investigate the cookie lifetime.
• When power-cycling the logging stops. There is a check on startup; when no connection can be made, the heartbeat stops (displaying repeating the text: Plugin not running. Please check parameters and LAN connection and restart plugin ). Then a restart of the plugin is necessary, but when the plugin was correctly running, the heartbeat should not stop!

[0crap]

...

• When I'm on D7 I'll take a look at username/password vs token. Or make both options available :-) and investigate the cookie lifetime.

Fair enough!

And the password is now masked with all dots, thx. All working fine, so I'll close this issue.

Kudos for fixing this one. :-)

[harolds]

This development version is working for me too. Only had to add the username and password and now all panels show up in Domoticz. Cool!