Add support for 6 month owner tokens through enlighten API, and batte…

[DanBeard]

This solves [gh-78] for my newly provisioned system on Firmware version D7.0.71 . I couldn't figure out how to get the token working from the 'entrez' api, but I found a technical brief by enphase on how to get token through enlighten that is good for 6 months. (link here)

This commit adds a new optional use_enlighten_owner_token param. If set it will attempt to log in to enlighten and grab the token from the enlighten API. I also added the ability to add a time buffer so it will refresh before the expiry date. This can be useful for offline use, so if you grab a new token every month you can be sure you'll have ~5 months left if enphase's server or your internet connection goes down.

It seems that the bearer token only works on the '/auth/check_jwt' endpoint. The other endpoints also want the session cookie that is returned by that endpoint or they return 401. So the change to add cookies to the client may also solve 401s with the entrez api tokens (but I couldn't test that because the entrez token generation api doesn't work for me for whatever reason)

I was also missing battery information for my system even though I have a battery backup. I found out from this post that you have to go through the 'ensemble' API for my model. (No idea how they figured that out. Is this API fully documented anywhere?)

I don't think I modified anything in the other data flows but I can't test on any systems but mine. Let me know if this causes any issues

Pull request recommendations:

• [ ] Name your pull request your-development-type/short-description. Ex: feature/read-tiff-files
• [ ] Link to any relevant issue in the PR description. Ex: Resolves [gh-12], adds tiff file format support
• [ ] Provide context of changes.
• [ ] Provide relevant tests for your feature or bug fix.
• [ ] Provide or update documentation for any feature added by your pull request.

Thanks for contributing!

[DanBeard]

Rebased on [gh-90] and expanded it to check if cookie can be refreshed. For this use case it's not that the token itself times out (that's good for 6 months), but that the session cookie used to talk to the envoy times out and needs to be refreshed locally.

I moved the PR to draft until I can confirm it's stable for a while. If you want to use these changes in HA you can use the custom component I made at https://github.com/DanBeard/enphase_envoy

edit: When paired with that custom component, I can see battery percentage too, so this may also solve [gh-72] ?

[michaelgaultjr]

Rebased on [gh-90] and expanded it to check if cookie can be refreshed. For this use case it's not that the token itself times out (that's good for 6 months), but that the session cookie used to talk to the envoy times out and needs to be refreshed locally.

I moved the PR to draft until I can confirm it's stable for a while. If you want to use these changes in HA you can use the custom component I made at https://github.com/DanBeard/enphase_envoy

edit: When paired with that custom component, I can see battery percentage too, so this may also solve [gh-72] ?

I have two combiners, one is running D5.x.x, the other is running D7.x.x, I am getting this error for both when pressing "Submit" on the setup config flow

Logger: custom_components.enphase_envoy.config_flow
Source: custom_components/enphase_envoy/config_flow.py:40
Integration: Enphase Envoy (DEV)
First occurred: 12:16:18 PM (2 occurrences)
Last logged: 12:17:59 PM

Unexpected exception
Traceback (most recent call last):
  File "/config/custom_components/enphase_envoy/config_flow.py", line 154, in async_step_user
    envoy_reader = await validate_input(self.hass, user_input)
  File "/config/custom_components/enphase_envoy/config_flow.py", line 40, in validate_input
    use_enlighten_owner_token=data[CONF_USE_ENLIGHTEN],
KeyError: 'use_enlighten'

I thought I had this installed and working fine yesterday, even did multiple restarts and reloads, however today without restarting or reloading the the integration disconnected from both combiners after first giving a 401 unauthorized at around 5am (I can't find the log entry) errors for just the combiner running D7.x.x.

Before using this integration I was using the one provided at https://github.com/jesserizzo/envoy_reader/issues/78#issuecomment-1049399793, it was working fine, however it would disconnect every morning at around 4-5am, which I'm assuming is related to not using the 6 month access token.

[DanBeard]

Thanks for trying it out @michaelgaultjr . I just fixed the crash when the checkbox is unchecked. If you want to test the 6 month token be sure the "use enlighten" checkbox is checked and the username and password are what you would use on enlighten.enphaseenergy.com .

Thanks for the link: I didn't know about that other integration. I'll try to rebase mine on it so we can have the best of both.

I've been running with my custom_component for ~5 days now (Firmware: D7.0.71) and it hasn't disconnected yet. Not sure how to adequately test the 6 month token refresh in a reasonable amount of time though.

[michaelgaultjr]

Thanks for trying it out @michaelgaultjr . I just fixed the crash when the checkbox is unchecked. If you want to test the 6 month token be sure the "use enlighten" checkbox is checked and the username and password are what you would use on enlighten.enphaseenergy.com .

Thanks for the link: I didn't know about that other integration. I'll try to rebase mine on it so we can have the best of both.

I've been running with my custom_component for ~5 days now (Firmware: D7.0.71) and it hasn't disconnected yet. Not sure how to adequately test the 6 month token refresh in a reasonable amount of time though.

Thanks for the quick fix and reply, those issues have been solved, however I'm now running into two different issues.

For the D7.x.x combiner, when pressing "Submit" in the config flow using the enphase enlighten email and password, with Use Enlighten checked` it says "Unexpected Error" and I get this error in the logs

Logger: custom_components.enphase_envoy.config_flow
Source: custom_components/enphase_envoy/config_flow.py:46
Integration: Enphase Envoy (DEV)
First occurred: 2:44:42 PM (1 occurrences)
Last logged: 2:44:42 PM

Unexpected exception
Traceback (most recent call last):
  File "/config/custom_components/enphase_envoy/config_flow.py", line 154, in async_step_user
    envoy_reader = await validate_input(self.hass, user_input)
  File "/config/custom_components/enphase_envoy/config_flow.py", line 46, in validate_input
    await envoy_reader.getData()
  File "/usr/local/lib/python3.9/site-packages/envoy_reader/envoy_reader.py", line 355, in getData
    await self._getEnphaseToken()
  File "/usr/local/lib/python3.9/site-packages/envoy_reader/envoy_reader.py", line 250, in _getEnphaseToken
    self._token = token_json["token"]
KeyError: 'token'

And for the D5.x.x combiner, the integration is added, however it doesn't load due to this error

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 187, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 147, in _async_update_data
    return await self.update_method()
  File "/config/custom_components/enphase_envoy/__init__.py", line 66, in async_update_data
    battery_data = await envoy_reader.battery_storage()
  File "/usr/local/lib/python3.9/site-packages/envoy_reader/envoy_reader.py", line 717, in battery_storage
    return self.endpoint_ensemble_json_results.json()[0]["devices"]
KeyError: 0

This seems to be because this combiner doesn't have any batteries connected to it, as our setup is a little strange due to the way the installer set them up, the D7.x.x combiner has 15 panels and the batteries, and the D5.x.x combiner just has ~45 panels.

[DanBeard]

Yeah, it didn't handle "ensemble" response without a battery well. I've added a guard around that.

I think the 'KeyError: 'token'" on the D7.x.x combiner is because your enlighten account doesn't have permission to access it by its serial number. If you login to enlighten.enphaseenergy.com and then in that same browser window go to https://enlighten.enphaseenergy.com/entrez-auth-token?serial_num=SERIAL_NUMBER_HERE with the serial number for the unit on firmware D7.x.x, does it give you an error message? I've pushed new changes that should catch a bad response here and make the exception more helpful at least.

If you want to try it out the new changes with the custom component, you may need to delete the envoy_reader directory and envoy_reader-0.21.3.dist-info in "/usr/local/lib/python3.9/site-packages" before using the custom component again.

[michaelgaultjr]

Yeah, it didn't handle "ensemble" response without a battery well. I've added a guard around that.

I think the 'KeyError: 'token'" on the D7.x.x combiner is because your enlighten account doesn't have permission to access it by its serial number. If you login to enlighten.enphaseenergy.com and then in that same browser window go to https://enlighten.enphaseenergy.com/entrez-auth-token?serial_num=SERIAL_NUMBER_HERE with the serial number for the unit on firmware D7.x.x, does it give you an error message? I've pushed new changes that should catch a bad response here and make the exception more helpful at least.

If you want to try it out the new changes with the custom component, you may need to delete the envoy_reader directory and envoy_reader-0.21.3.dist-info in "/usr/local/lib/python3.9/site-packages" before using the custom component again.

Thanks for fixing the ensemble issue. I'm not home to install the new version of the plugin, but I was able to check the token using that link, I didn't get any errors.

[michaelgaultjr]

Finally got a chance to try it out. We had some changes made to our system, so D5.x.x combiner (the one with all the solar panels) has been updated to D7.0.71, and with the update to the integration it seems to be working, though I haven't been using it for long.

One thing I'm noticing is there's no way to get the amount of energy going in and out of the batteries, just the current charge percentage, It'd be really nice to have those so I can track energy in Home Assistant instead of the Enphase cloud. I'm used this guide this guide on the forum to setup grid import and export sensors, however this doesn't work with the batteries without knowing how much is going in and out.

[DanBeard]

Yeah I want to calculate the same thing, but It doesn't look like the "ensemble" api gives the production/consumption from the battery itself.

Here's an example of the json that the api gives:

[ { "type": "ENCHARGE", "devices": [ { "part_num": "redacted", "installed": 0, "serial_num": "redacted", "device_status": [ "envoy.global.ok", "prop.done" ], "last_rpt_date": 1647635509, "admin_state": 6, "admin_state_str": "ENCHG_STATE_READY", "created_date": 1647347030, "img_load_date": 1647347030, "img_pnum_running": "2.0.4336_rel/21.19", "zigbee_dongle_fw_version": "100D", "bmu_fw_version": "2.1.16", "operating": true, "communicating": true, "sleep_enabled": false, "percentFull": 100, "temperature": 24, "maxCellTemp": 25, "comm_level_sub_ghz": 4, "comm_level_2_4_ghz": 4, "led_status": 14, "dc_switch_off": false, "encharge_rev": 255, "encharge_capacity": 3500 }, { "part_num": "redacted", "installed": 0, "serial_num": "redacted", "device_status": [ "envoy.global.ok", "prop.done" ], "last_rpt_date": 1647635497, "admin_state": 6, "admin_state_str": "ENCHG_STATE_READY", "created_date": 1647347044, "img_load_date": 1647347044, "img_pnum_running": "2.0.4336_rel/21.19", "zigbee_dongle_fw_version": "100D", "bmu_fw_version": "2.1.16", "operating": true, "communicating": true, "sleep_enabled": false, "percentFull": 100, "temperature": 25, "maxCellTemp": 25, "comm_level_sub_ghz": 5, "comm_level_2_4_ghz": 5, "led_status": 14, "dc_switch_off": false, "encharge_rev": 255, "encharge_capacity": 3500 }, { "part_num": "redacted", "installed": 0, "serial_num": "redacted", "device_status": [ "envoy.global.ok", "prop.done" ], "last_rpt_date": 1647635515, "admin_state": 6, "admin_state_str": "ENCHG_STATE_READY", "created_date": 1647347046, "img_load_date": 1647347046, "img_pnum_running": "2.0.4336_rel/21.19", "zigbee_dongle_fw_version": "100D", "bmu_fw_version": "2.1.16", "operating": true, "communicating": true, "sleep_enabled": false, "percentFull": 100, "temperature": 25, "maxCellTemp": 26, "comm_level_sub_ghz": 5, "comm_level_2_4_ghz": 5, "led_status": 14, "dc_switch_off": false, "encharge_rev": 255, "encharge_capacity": 3500 } ] }, { "type": "ENPOWER", "devices": [ { "part_num": "redacted", "installed": 0, "serial_num": "redacted", "device_status": [ "envoy.global.ok", "prop.done" ], "last_rpt_date": 1647635560, "admin_state": 24, "admin_state_str": "ENPWR_STATE_OPER_CLOSED", "created_date": 1647347037, "img_load_date": 1647347037, "img_pnum_running": "1.5.3637_rel/21.19", "zigbee_dongle_fw_version": "100D", "operating": true, "communicating": true, "temperature": 82, "comm_level_sub_ghz": 5, "comm_level_2_4_ghz": 5, "mains_admin_state": "closed", "mains_oper_state": "closed", "Enpwr_grid_mode": "multimode-ongrid", "Enchg_grid_mode": "multimode-ongrid", "Enpwr_relay_state_bm": 3376, "Enpwr_curr_state_id": 16 } ] } ]

[michaelgaultjr]

Yeah I want to calculate the same thing, but It doesn't look like the "ensemble" api gives the production/consumption from the battery itself.

Here's an example of the json that the api gives:

[ { "type": "ENCHARGE", "devices": [ { "part_num": "redacted", "installed": 0, "serial_num": "redacted", "device_status": [ "envoy.global.ok", "prop.done" ], "last_rpt_date": 1647635509, "admin_state": 6, "admin_state_str": "ENCHG_STATE_READY", "created_date": 1647347030, "img_load_date": 1647347030, "img_pnum_running": "2.0.4336_rel/21.19", "zigbee_dongle_fw_version": "100D", "bmu_fw_version": "2.1.16", "operating": true, "communicating": true, "sleep_enabled": false, "percentFull": 100, "temperature": 24, "maxCellTemp": 25, "comm_level_sub_ghz": 4, "comm_level_2_4_ghz": 4, "led_status": 14, "dc_switch_off": false, "encharge_rev": 255, "encharge_capacity": 3500 }, { "part_num": "redacted", "installed": 0, "serial_num": "redacted", "device_status": [ "envoy.global.ok", "prop.done" ], "last_rpt_date": 1647635497, "admin_state": 6, "admin_state_str": "ENCHG_STATE_READY", "created_date": 1647347044, "img_load_date": 1647347044, "img_pnum_running": "2.0.4336_rel/21.19", "zigbee_dongle_fw_version": "100D", "bmu_fw_version": "2.1.16", "operating": true, "communicating": true, "sleep_enabled": false, "percentFull": 100, "temperature": 25, "maxCellTemp": 25, "comm_level_sub_ghz": 5, "comm_level_2_4_ghz": 5, "led_status": 14, "dc_switch_off": false, "encharge_rev": 255, "encharge_capacity": 3500 }, { "part_num": "redacted", "installed": 0, "serial_num": "redacted", "device_status": [ "envoy.global.ok", "prop.done" ], "last_rpt_date": 1647635515, "admin_state": 6, "admin_state_str": "ENCHG_STATE_READY", "created_date": 1647347046, "img_load_date": 1647347046, "img_pnum_running": "2.0.4336_rel/21.19", "zigbee_dongle_fw_version": "100D", "bmu_fw_version": "2.1.16", "operating": true, "communicating": true, "sleep_enabled": false, "percentFull": 100, "temperature": 25, "maxCellTemp": 26, "comm_level_sub_ghz": 5, "comm_level_2_4_ghz": 5, "led_status": 14, "dc_switch_off": false, "encharge_rev": 255, "encharge_capacity": 3500 } ] }, { "type": "ENPOWER", "devices": [ { "part_num": "redacted", "installed": 0, "serial_num": "redacted", "device_status": [ "envoy.global.ok", "prop.done" ], "last_rpt_date": 1647635560, "admin_state": 24, "admin_state_str": "ENPWR_STATE_OPER_CLOSED", "created_date": 1647347037, "img_load_date": 1647347037, "img_pnum_running": "1.5.3637_rel/21.19", "zigbee_dongle_fw_version": "100D", "operating": true, "communicating": true, "temperature": 82, "comm_level_sub_ghz": 5, "comm_level_2_4_ghz": 5, "mains_admin_state": "closed", "mains_oper_state": "closed", "Enpwr_grid_mode": "multimode-ongrid", "Enchg_grid_mode": "multimode-ongrid", "Enpwr_relay_state_bm": 3376, "Enpwr_curr_state_id": 16 } ] } ]

Yeah, I ended up digging around a bunch and found out there is no way to get it aside from their cloud API, I read somewhere local battery information was planned for we 2021, but that unfortunately hasn't happened

[jesserizzo]

@DanBeard Have you or anyone else checked this on older firmwares?