Doesn’t work on Envoy 7.x.x firmware

immesys commented 2 years ago

I am getting 401 Authorization Required when the requests gets made to the stream endpoint.

I can reproduce with curl as well (installer password is generated using envoy serial number in the script you provided):

Doing this

curl --insecure --digest --user installer:123456 https://192.168.1.115/stream/meter

Gives me back this:


<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>Redirecting to <a id='link' href=''></a></center>
<script>
const redirect_url = window.location.protocol + '//' + window.location.host + '/home';
function redirect() { window.location.href = redirect_url; }
window.onload = function() {
    const link = document.getElementById('link');
    link.href = redirect_url;
    link.text = redirect_url;
    setTimeout(redirect, 500);
}
</script>
</body>
</html>

My envoy is running firmware D7.0.66 (ddd0d7).

Is your envoy running the same firmware, but it works? Do you not have issues with getting redirected to HTTPS and having a self-signed certificate cause issues? I noticed you have modified your code recently, so I assumed you are still using this code successfully.

niggles1973 commented 1 year ago

I've got an early flight; so need to stop looking; one for the weekend when I'm back home - that process for the token generation is the one I used - I got my token this last weekend - can I confirm is your account setup as an owner and not an installer? my account is an owner account (token has a 1 year expiry)

helderfmf commented 1 year ago

All this endpoints I got from this post https://community.home-assistant.io/t/enphase-envoy-current-state-of-things/443829/155 that links this doc https://1drv.ms/b/s!AjADZOjtW-fdr_9AGH_IGqyxMg0Fkg?e=Rc98rf I don't know where to confirm if I'm an owner or an installer, but my system is managed by the company that installed it, I asked for access, and they gave me a link to register. When I go to https://enlighten.enphaseenergy.com/ I'm not listed as the installer but the company that installed it is.

So my token must be of Owner and not Installer.

helderfmf commented 1 year ago

If it was a proper stream, it should say:

< Content-Type: text/event-stream

But it's fast, and gives the sum of production, grid, net-consumption and batteries. Even if we have to query each time, instead of only "listening" as with a stream.

del13r commented 1 year ago

I tested the difference between https://envoy.local/ivp/livedata/status And https://envoy.local/ivp/meters/readings

they are both json output with equal speed and latency and both have to be queried.

The difference being that ivp/meters/readings appears to give live power and lifetime energy odometer stats of the 2 ct clamps. 1 for panels/production 1 for net grid ( when number is positive this means grid import, and when negative this means grid export).

whereas with ivp/livedata/status, you have to enable a streaming flag first which is misleading as it doesn’t appear to be actually streaming anywhere if you have to repeatedly query it. I don’t currently have batteries so my interest in this url is lower as it doesn’t provide me any extra.

I’ve written an article here

https://community.home-assistant.io/t/enphase-envoy-with-energy-dashboard/328668/662?u=del13r

helderfmf commented 1 year ago

Just a heads up, tested https://envoy.local/ivp/meters/readings, and its as fast as the livedata, so implemented in my fork, also the automatic token generation on startup and in the case of expired token. Sofar its running stable, getting readings in 1s frequency. Animation

Animation

del13r commented 1 year ago

Great effort. Much appreciated.

Just wondering, is your envoy connected to wifi or Ethernet? I ask this as I had an issue/bug this week where my envoy repeatedly kept trying to disassociate and then Re-associate to my router several times a minute via wifi. I restarted my envoy and that didn’t stop the issue, so I restarted my router and that seemed to fix it. So I’m unsure as to why my router and the envoy weren’t getting along at that point in time. I assumed it had something to do with how fast I was querying the envoy. Something to keep an eye out for anyway.

helderfmf commented 1 year ago

Had it on wifi but sometimes got packet loss, wifi signal on envoy is very bad. Next to it my phone had 3 of 4 bars and envoy was missing packets and disconnect from wifi. Got a Wifi repeater and installed 2m distance from it, but even then it would miss some reports. I have an automation in HA that notifies me if the sensor is not updated for more than 45s. Gave up and installed the repeater next to it and passed a Ethernet cable. Problem solved. This was all with FW 5, don't know if FW 7 improves anything. I think it's more of a hardware problem, not a software one.

vk2him commented 1 year ago

Excellent work on this solution - well done! I "may" modify my code to have an initial test to determine the Envoy firmware - I could then use my original code for v5 and yours for v7 .

del13r commented 1 year ago

Yep, even at 15 seconds, my wifi envoy and router arent getting along right occasionally.

Syslog messages on my router: 13:28:07 eth1: Deauth_ind 64:33:DB:HIDDEN, status: 0, reason: Disassociated due to inactivity (4), rssi:0 13:29:14 eth1: Auth 64:33:DB:HIDDEN, status: Successful (0), rssi:0 13:29:14 eth1: Auth 64:33:DB:HIDDEN, status: Successful (0), rssi:0 13:29:14 eth1: Assoc 64:33:DB:HIDDEN, status: Successful (0), rssi:0 13:29:14 eth1: Assoc 64:33:DB:HIDDEN, status: Successful (0), rssi:0 13:29:14 eth1: Assoc 64:33:DB:HIDDEN, status: Successful (0), rssi:0 13:29:18 eth1: Auth 64:33:DB:HIDDEN, status: Successful (0), rssi:0 13:29:18 eth1: Assoc 64:33:DB:HIDDEN, status: Successful (0), rssi:0 13:30:27 eth1: Deauth_ind 64:33:DB:HIDDEN, status: 0, reason: Disassociated due to inactivity (4), rssi:0 13:33:23 eth1: Auth 64:33:DB:HIDDEN, status: Successful (0), rssi:0 13:33:24 eth1: Assoc 64:33:DB:HIDDEN, status: Successful (0), rssi:0

I'm blaming the router at the moment as the problem stopped after this message in the syslog on the router: 13:33:59 kernel: nvram: consolidating space!

Perhaps I can potentially query the envoy more often and the problem was my router all along.

vk2him commented 1 year ago

What router do you have? I had an Asus that worked fine but I've replaced a while ago and now run OPNsense

del13r commented 1 year ago

I have a 10 year old Asus Router running merlin firmware. I think I am overloading the router CPU and/or MEMORY by running too many features at once on this old router like:

diversion ad blocker
connmon to healthchecks.io
openvpn server
DNS-over-TLS (DoT)

I've now disabled some of these features to see if that improves my drop outs.

vk2him commented 1 year ago

I had the exact same setup on an Asus RT-AC86U with Merlin and the same (and extra) features. I got frequent issues that I attributed to memory leaks as I found once a week I had to physically power it off/on as the schedule reboot didn't help. Thats basically why I changed to OPNSense running on a Protectlic Small form factor "brick" PC - it does all those things (and more) . Instead of diversion I use Adguard home plus deep packet inspection etc. I still use the Asus but I configured them in AP Mode as a Mesh network (with 2 x AC68U). Very stable now

helderfmf commented 1 year ago

@vk2him can you query your envoy with curl http://envoy.local/info to see if in FW v5 it responds with XML with some information of the Envoy. That could be a way to automatically chose which method to use on the python script. One of the tags is the firmware version.

vk2him commented 1 year ago

@helderfmf - thanks for the shoutout

Yes that url provides a nice xml - here's the snippet for the firmware. Should be easy to use that to select the v5/v7 code

<software>D5.0.62</software>

Thanks

helderfmf commented 1 year ago

@vk2him Give my version a spin:

Compatible with both FW 5 and 7 with auto detection of version
FW 5 auto generate of installer password
FW 7 auto generate of token

vk2him commented 1 year ago

@helderfmf I tried your version - it didn't start as it seems you're trying to connect to the envoy via https - my V5 firmware only allows http access. Here's the error, maybe change to http port 80 for v5?


    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.1.84', port=443): Max retries exceeded with url: /info (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f58b0f66d50>: Failed to establish a new connection: [Errno 111] Connection refused'))```

helderfmf commented 1 year ago

Quick fix done. That url is available in both https and http in fw7. So was only a small change to downgrade to http.

vk2him commented 1 year ago

I have your version running and I can see the data in mqttexplorer however you've changed the mqtt_topic to the probably more correct value and I'll need to change all my templates to test it. You have:

MQTT_TOPIC = "envoy/json"

My version has a / in front:

MQTT_TOPIC = "/envoy/json"

Do you think you could add a configuration option in your version to allow changing the mqtt_topic? Cheers

vk2him commented 1 year ago

I changed the templates as it was easy and I did a quick reload of configuration.yaml - it seems to be working well! I'll monitor for a while and give you feedback. Might still be an idea to allow people to change the topic like you're doing with freeds?

Would you be Ok if I merged your change into mine? I'll also update the documentation with info for v5/v7 however since I don't have V7, maybe you could give some info on what the stream looks like and any templates needed to extract the 1 second data? Thanks

helderfmf commented 1 year ago

Sure, that was the purpose of my fork, to accommodate my needs and to improve the original script. Already updated manual with some information., and with your suggestion, just make sure you are using the latest version 1.0.9.15, had some bugs with lower versions. Just didn't include the information of the new parameters of the addon. When you do please include a reference to this project https://github.com/pablozg/freeds/ , that's the one responsible for inclusion of the MQTT_TOPIC_FREEDS. And why not, go and check it out, it can use the excess of power production to adjust the power of a boiler or any "dumb" heater appliance that has a resistor.

vk2him commented 1 year ago

Thanks for that Helder - I'll check out freeds because I have an electric hot water system that could use this :)

del13r commented 1 year ago

I have an appliance that works the same as freeds, just a little less controllable out of the box.

Device https://www.paladinsolarcontroller.com.au/product/solar-power-diverter-controller/

User Manual https://www.paladinsolarcontroller.com.au/wp-content/uploads/2019/03/Paladin-V6-User-Guide.pdf

How I got a little more control over the appliance: I have wired a SONOFF RE5V1C to the solid state relay to do 2 things using either the timer function or home assistant automation:

Prevent all power transfer to the hot water heater in peak times when its most expensive.
Allow power transfer when power is power is free from the solar panels or cheapest (offpeak rate).

Detail: I was able to use the 5v and -V from the paladin board to power the 5V sonoff.

Then I located the yellow wire going from terminal 3 of the solid state relay (SSR) back to the paladin board. I cut the yellow wire in the middle and soldered in the sonoff to both cut ends of the yellow wire.

vk2him commented 1 year ago

@del13r - very nice! A few years ago I had a similar device that had an inbuilt sensor which would automatically divert any excess solar generated into the Hot Water heater. It had a similar price as well to yours. If the water was up to temperature it would then sell excess solar. It worked really well, sub-second response and it saved me a lot of money - initially. One day it went BANG in a big way, the SCRs completely exploded and nearly caught fire. The Australian distributor that installed it had gone out of business so it took over 9 months to get a warranty replacement one from UK. In the mean time, the cost of Off Peak had reduced below my feed in tariff, so it was cheaper for me to export excess power and buy off peak, so I sold it. It's still that way today, I get $20c FIT but I'm expecting that to end in a few months so Off peak will cost more. However the difference in cost will be a few cents, so it will be a very long time to get a return on investment if I buy a new device. Still, it was very cool to watch the 1 second power generation/consumption and see how quickly the excess power went to the Hot water and then stopped as a cloud covered the panels. https://www.immersun.co.uk/products/power-diverter

helderfmf commented 1 year ago

Both of your products are commercial grade ones, and because of that it comes with a premium price, and of course warranty, support, assurance, and others features. FREEDS is a DIY approach, with a BOM of about 50€. Since I lack proper soldering skills, I bought one fully assembled for 100€. In my country we have a net-metering of 15 minutes, it means that every 15 minutes the utility meter resets and every W we inject on the grid is subtracted with every W we consume. It combines well with the way FREEDS works, since it doesn't have an internal meter for the grid, and for the output, it relies on 3rd party meters, my case the envoy, and utilizes a PID to calculate the % of PWM to deliver to output, because of that it cannot guarantee 1s 0 export, it takes more seconds to get there. But it works very well: Screenshot_2023-07-16-10-14-21-009-edit_io.homeassistant.companion.android.jpg

Screenshot_2023-07-16-10-14-39-271-edit_io.homeassistant.companion.android.jpg

vk2him commented 1 year ago

Reopening to merge @helderfmf version - thanks Helder!

helderfmf commented 1 year ago

Just another heads up, I don't know why, but with this version every time I restart HA it stops reporting new values, I have to restart add-on and then is perfect again. Don't know if it is something with my MQTT sensors, or the python script in the add-on, or even FW 7, since there is no log of error in the timeframe of before and after reboot. I worked around this with an automation to restart add-on after startup of HA, it's ugly but effective. Maybe put this warning in the description.

vk2him commented 1 year ago

I don't think it's your or my code as I've also seen this strange mqtt behaviour for the last few HA releases after a HA restart. I also noticed other addons don't always start after a HA restart and there's no pattern to it. This only started a month or so ago with a HA monthly version update ...

It affects all my other (non envoy) mqtt integrations and sensors as they don't work after a restart. I found sometimes I needed to restart the mqtt broker and other times restart the addons ... as a test I might disable the envoy mqtt addon and see if the others start properly.

vk2him commented 1 year ago

@helderfmf - regarding your version - do you cache the enphase token? I'm wondering what happens if the internet is down when you restart HA - will your v7 mods fail if it can't get a token or will it use a cached token? I'll wait your reply before I publish an updated version. Thanks

thbrd commented 1 year ago

Hi @vk2him & @helderfmf I found this thread/fork last couple days and I am glad you fixed the add-on for version 7. But I get an error when I start this add-on. This is my log:

s6-rc: info: service s6rc-oneshot-runner: starting s6-rc: info: service s6rc-oneshot-runner successfully started s6-rc: info: service fix-attrs: starting s6-rc: info: service fix-attrs successfully started s6-rc: info: service legacy-cont-init: starting s6-rc: info: service legacy-cont-init successfully started s6-rc: info: service legacy-services: starting s6-rc: info: service legacy-services successfully started 19/07/2023 09:27:11 Serial number: [serial number] 19/07/2023 09:27:11 Detected FW version 7 19/07/2023 09:27:11 No token avaliable, generating one 19/07/2023 09:27:11 Connected to core-mosquitto:1883 Exception in thread Thread-2 (scrape_stream_meters): Traceback (most recent call last): File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner self.run() File "/usr/lib/python3.11/threading.py", line 975, in run self._target(*self._args, **self._kwargs) File "//envoy_to_mqtt_json.py", line 324, in scrape_stream_meters json_string_freeds = json.dumps(round(stream.json()[1]["activePower"]))


IndexError: list index out of range
envoy/json
19/07/2023 09:27:11 Token generated [token is here]
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

I have removed the token and the serial, My MQTT is working perfectly. 
Did I made a mistake or something else? 

My envoy version is: D7.6.172
I am also running "Enphase Envoy (Installer)" This allows me to enable and disable the solar panels. But the states are not live like when with MQTT.

vk2him commented 1 year ago

Hi @thbrd - can you please confirm if you're using my newly published version or the one from @helderfmf ? FYI - today I forked his version and published it (temporarily for testing ) under https://github.com/vk2him/Enphase-Envoy-mqtt-json-v7

I don't have a v7 Envoy so I'm unable to test the v7 side of it - I'll need @helderfmf to advise us on this. Thanks

thbrd commented 1 year ago

@vk2him Ofcourse :)
Helders version Running: Current version: 1.0.9.15 from this add-on location: https://github.com/helderfmf/Enphase-Envoy-mqtt-json

helderfmf commented 1 year ago

@helderfmf - regarding your version - do you cache the enphase token? I'm wondering what happens if the internet is down when you restart HA - will your v7 mods fail if it can't get a token or will it use a cached token? I'll wait your reply before I publish an updated version. Thanks

True, I don't cache the token anywhere, if there isn't internet it will fail miserably. We could:

create a configuration item in the add-on to manual input the token, and then read it at startup as with the other options
cache in a file the last token generated, read it in the startup if no configuration item is read.

vk2him commented 1 year ago

True, I don't cache the token anywhere, if there isn't internet it will fail miserably. We could:

create a configuration item in the add-on to manual input the token, and then read it at startup as with the other options

cache in a file the last token generated, read it in the startup if no configuration item is read.

I think option 2 is best but not to allow manually entering the token in configuration. I've read on of the extremely long Enphase forum posts that people are complaining a lot that when their internet goes down, which happens if they have a power failure for example and they don't have a ups on their router/modem, that they can't access the envoy as they need the token. If you could store the last valid token in a file and use that first that would be best - I mention not entering this into a config item because if they forget to update it, the token will potentially have expired.

helderfmf commented 1 year ago

@thbrd by the log seams it could not read the json output from the envoy. Can you post the output of the following command, just replace YourVeryLongToken for the token you got from the add-on.

curl -k -H "Authorization: Bearer YourVeryLongToken" https://envoy.local/ivp/meters/readings

thbrd commented 1 year ago

@helderfmf Seems to be empty. I only see []

When I go to the URL "/api/v1/production" I will see this: { "wattHoursToday": 6022, "wattHoursSevenDays": 167760, "wattHoursLifetime": 173817, "wattsNow": 2459 }

Maybe this will help?

helderfmf commented 1 year ago

can you try this one: curl -k -H "Authorization: Bearer YourVeryLongToken" https://envoy.local/production.json

thbrd commented 1 year ago

@helderfmf Yes working

{"production":[{"type":"inverters","activeCount":13,"readingTime":1689764459,"wNow":1993,"whLifetime":174013}],"storage":[{"type":"acb","activeCount":0,"readingTime":0,"wNow":0,"whNow":0,"state":"idle"}]}

thbrd commented 1 year ago

Maybe we can add the function to select which page to use from the Envoy? With that selection different json variables will be used for mqtt?

vk2him commented 1 year ago

I had thought about adding the option to change which page to use, but decided not to for two reasons 1) It's super easy for anyone to fork this code and make the change themselves 2) Too many people will add the wrong data and then ask for support, or put in the right data and then need help with the value_template to extract the data in HA

Basically if you're wanting to use different pages which will require different Home assistant value_templates (which can be tricky), you should already have the skills to fork and do it yourself.

thbrd commented 1 year ago

I have forked it and will give it a try. My first Github project haha

vk2him commented 1 year ago

Should be easy as I said - change the envoy page in envoy_to_mqtt_json.py also change line 25 the readme.md file to point to your forked github directory, that way the blue Add Repository buton will add it to your addons. Then modify the sensor value_template to match the new data feed. It helps to install mqttexplorer to help visualise the data https://mqtt-explorer.com/

helderfmf commented 1 year ago

Already have everything you need. The function is there, only need to change the main function to call the stream_production function.

thbrd commented 1 year ago

Thanks! will try to fix it for my Envoy setup :)

helderfmf commented 1 year ago

new version up, with persistent token. 1.0.9.18

thbrd commented 1 year ago

@helderfmf thank you for the work and preparation (line 555) I have one update on your script. On line 287 you need to change "consumption" to "production".

Now the script is sending data to MQTT and got it working! Thank you

mqtt: sensor:

name: envoy mqtt production state_topic: "envoy/json" value_template: "{{ value_json.production[0].wNow }}" unique_id: envoy_mqtt_production qos: 0 unit_of_measurement: "W" state_class: measurement device_class: power

helderfmf commented 1 year ago

I'm sorry but i updated the file and maybe the numbers don't match up, but your line 287 is the line with the topic for freeds, correct? If so, that is correct, at least for me, I'm trying to send consumption to FREEDS not production. You don't need that for HA, as the entire JSON is sent via MQTT. If you are not using FREEDS, then in your config of the add-on just remove the default FREEDS_TOPIC.

thbrd commented 1 year ago

Ah okay, received an error on that line with consumption when I tried it with production (what I have in my json string) it is working.

{"production":[{"type":"inverters","activeCount":13,"readingTime":1689771174,"wNow":1584,"whLifetime":178613}],"storage":[{"type":"acb","activeCount":0,"readingTime":0,"wNow":0,"whNow":0,"state":"idle"}]}

Now the add-on is working and I see in the MQTT explorer the whole production.json string. Like above.

helderfmf commented 1 year ago

@vk2him when you have time, please test this new version with persistent token, removed topic of FREEDS, since maybe only me uses, created a toggle to activate it. Also implemented a lot of error catching of the JSON responses, so it will not crash that easily, and gives some meaningful output. Tested all the mods of FW 7, but cannot make any debug from FM 5. version 19.

vk2him commented 1 year ago

@helderfmf - yes will do. I've also just learned that @del13r has discovered that his v7 envoy is now streaming /stream/meter - I wonder if you can see that as well? Is so, I'll probably use that in my version of your code to be consistent across v5/v7. This is his envoy details

Software Version D7.6.175 (f79c8d) Software Build Date 22 Jun, 2023 8:43 PM

https://envoy.local/stream/meter

vk2him commented 1 year ago

@vk2him when you have time, please test this new version with persistent token, removed topic of FREEDS, since maybe only me uses, created a toggle to activate it. Also implemented a lot of error catching of the JSON responses, so it will not crash that easily, and gives some meaningful output. Tested all the mods of FW 7, but cannot make any debug from FM 5. version 19.

@helderfmf - I tested the new version and saw two issues on FW5

1) The addon now has a Boolean slide switch for freeds that is enabled by default. You also added this to config.yaml: USE_FREEDS: Activate publish second topic to FREEDS - note this isn't boolean value

This causes the addon to abort with an error that the configuration expected a Boolean value error. The "fix" I think would be if you can have use freeds switch disabled by default?

2) Looks like the new version doesn't generate the FW5 password any more? Here's the error:

20/07/2023 09:47:25 Detected FW version 5 20/07/2023 09:47:25 FREEDS is inactive 20/07/2023 09:47:25 No token in file: /data/token.txt 20/07/2023 09:47:25 Generating new token 20/07/2023 09:47:25 Token generated LongEnphaseToken 20/07/2023 09:47:25 Connected to 192.168.1.74:1883 20/07/2023 09:47:25 Subscribed to MQTT_TOPIC: envoy/json self._target(*self._args, **self._kwargs) File "//envoy_to_mqtt_json.py", line 521, in scrape_stream if ENVOY_PASSWORD =='' or ENVOY_PASSWORD == None : ENVOY_PASSWORD=emupwGetMobilePasswd(serial, userName) ^^^^^^^^^^^^^^ NameError: name 'ENVOY_PASSWORD' is not defined. Did you mean: 'MQTT_PASSWORD'?

vk2him / Enphase-Envoy-mqtt-json

Doesn’t work on Envoy 7.x.x firmware #5