Open krconv opened 3 years ago
@krconv If you were using the dev branch, this might explain why your OTA wasn't working : https://github.com/esphome/esphome/pull/2852
I was flashing using the released version; trying to get a few things done this morning:
Just a FYI : https://github.com/esphome/esphome/pull/2871
I think we should have this merged before implementing new sensors and features (like calibration mechanism). The maintainers seem to prefer smaller PRs so it should be easier to merge other features afterwards.
Thanks for the hard work, I've been reading the progress here, and it's incredible what the three of you have got going! I picked up an emporia vue last week and I am yet to install it. I don't have anything useful to contribute towards the reverse engineering efforts, but I can help beta (alpha?) test your changes, if you are looking for volunteers.
I second what nightweyr said! I've been following along with this too, but I'm not sure that I have the skills to help with reverse engineering, but I could see what I could do.
I also just ordered an emporia vue and plan on holding off on installing it until there's hopefully something I can flash to it prior to it connecting anywhere else.
Lmk if there's anything that needs to be done next that I might be able to help with!
Hey @nightweyr, @williamobrien, glad to have you here!
I've prepared some instructions on getting this thing installed here: https://gist.github.com/flaviut/93a1212c7b165c7674693a45ad52c512
I'd appreciate hearing any questions & comments you may have, so that I can improve the document!
@flaviut Thanks for the detailed instructions, they are much nicer than I was expecting.
I am running into one tiny problem. If I set this as per the doc:
esp32:
board: d-duino-32
framework:
type: esp-idf
This block complains that This feature is only available with frameworks ['arduino']
esphome:
name: sub-panel
If I set framework to arduino, esphome complains that the emporia_vue platform is only available for framework: esp-idf.
I am on esphome-stable 2021.12.1. I do recall @Maelstrom96 submitting a few patches to core for a few bugfixes. Should I switch to the dev branch for this? I'll spin up a docker container on another host to test this - my main esphome is running on my home assistant box, and I am wary of switching that over to dev.
I see that your board had headers soldered on the breakout pins for serial, power and GPIO0. I have two of them that I purchased last month which didn't have them. I don't think this should be a blocker for anyone willing to open their vue and flash alternate firmware.
@nightweyr I'm also on 2021.12.1, and I don't get that error when I try and compile it. Would you mind posting the whole error message?
I see that your board had headers soldered on the breakout pins for serial, power and GPIO0. I have two of them that I purchased last month which didn't have them. I don't think this should be a blocker for anyone willing to open their vue and flash alternate firmware.
Yup, I soldered them on there since development is a lot longer than using something pre-developed. Definitely shouldn't be a problem for people who don't have a soldering iron, since they can just hold the jumpers in place for the minutes that this takes.
@nightweyr I'm also on 2021.12.1, and I don't get that error when I try and compile it. Would you mind posting the whole error message?
Same here, might be a good idea to try and still try and compile the code.
@flaviut Also, in your guide, I would change the backup command to esptool.py -b 921600 read_flash 0 0x800000 flash_contents.bin
instead. This will backup the whole flash instead of just where the runtime code resides. Also, setting the baud rate will significantly speed up the read_flash operation.
@Maelstrom96 My understanding is that the ESP32-WROVER-B has 4 MiB of flash, is there something else I'm missing here? Good idea on the baud rate though!
@flaviut The ESP32-WROVER-B doesn't have onboard flash and instead relies on flash is added by the integrator. In this case, in the logs, it was telling me that 8MB (64MBit) flash was detected when doing a read or flash.
I (40) boot: Enabling RNG early entropy source...
I (44) boot: SPI Speed : 40MHz
I (48) boot: SPI Mode : DIO
I (52) boot: SPI Flash Size : 8MB
I (670) psram: This chip is ESP32-D0WD
Huh, neat, guess that's the case. To be nitpicky, unless I'm missing something else, the datasheet specifies 4MiB flash unless it is a custom order (sec 1), but I failed to consider that they're using external flash.
Thanks for the correction, I've fixed it!
Oh, I wasn't sure why I thought that there wasn't any internal flash until I check the spec sheet for the ESP32-D0DW
, which specifies that there is no integrated flash. It is true that this particular ESP32 package has an "external flash" on the same board, and I guess they custom ordered them to have 8MB.
@nightweyr I'm also on 2021.12.1, and I don't get that error when I try and compile it. Would you mind posting the whole error message? With my usual template:
esphome: name: sub-panel
external_components:
esp32: board: d-duino-32 framework: type: esp-idf version: recommended
wifi: ssid: !secret wifi_ssid password: !secret wifi_password domain: !secret wifi_domain
ap: ssid: "$devicename Fallback Hotspot" password: !secret fallback_pw
logger:
web_server: ota: false
esp32_improv: authorizer: none
improv_serial:
api: password: !secret api_pw
ota: password: !secret ota_pw
I get this error:
root@15ef4d2f-esphome:/# esphome -v compile /config/esphome/sub-panel.yaml INFO Reading configuration /config/esphome/sub-panel.yaml... Failed config
json: None
This feature is only available with frameworks ['arduino']. {} async_tcp: None
This feature is only available with frameworks ['arduino']. {}
If I comment out the `framework` part of the yaml, I get this (which, to be clear, should be expected):
root@15ef4d2f-esphome:/# esphome -v compile /config/esphome/sub-panel.yaml INFO Reading configuration /config/esphome/sub-panel.yaml... Failed config
sensor.emporia_vue: [source /config/esphome/sub-panel.yaml:58]
This feature is only available with frameworks ['esp-idf']. platform: emporia_vue i2c_id: i2c_a phases:
Yup, I soldered them on there since development is a lot longer than using something pre-developed. Definitely shouldn't be a problem for people who don't have a soldering iron, since they can just hold the jumpers in place for the minutes that this takes.
Ha, I should have guessed that. I do have a few alligator clips to dupont jumpers at home. Let me try searching tor where they are, and for scientific reasons, let me see if I can clip them on steadily enough to complete the flash.
@nightweyr
Just tested your exact config and I got the exact same validation errors as you. Upon checking it and seeing if removing components would remove the validation warning, I found that removing the web_server
configuration was removing the validation error. Might be worth it to open a ESPHome issue since the validation error message and in the UI is very misleading in this particular case.
Mind me asking why you had that configuration
web_server:
ota: false
while still having set
ota:
password: !secret ota_pw
Edit: Just checked and that web_server setting as nothing to do with normal OTA, so nvm my question. But I think web_server is not supported with esp_idf. Good news tho, we should be able to use the Arduino library with the emporia_vue component soon since we were able to have the library fixed for our use case. I will have to add some code and test that everything works, but it should come eventually.
Duh, thanks! Sorry for the false alarm. I took out the webserver section and was able to compile the binary without any issues. The reason I didn't suspect it was because of the esphome error message, I'll submit a bug report there after testing a few more conditions. It's too late here to continue, but I hope to wrap this up this week and install it during the weekend.
Wow, those are some great instructions - thanks for putting that together! I'm just getting a chance to look at this right now and hopefully, in the next couple of days I'll have some time to give this a try.
One quick question - looking at the configuration, where would we enter the multiplier? For example, for a double pole breaker, would it be something like this?
- name: "2"
phase_id: phase_a
input: "2"
multiplier: "2"
Good question @williamobrien, I'll need to update the guide.
But there's https://esphome.io/components/sensor/index.html#calibrate-linear and I think that should take care of it!
Good question @williamobrien, I'll need to update the guide.
But there's https://esphome.io/components/sensor/index.html#calibrate-linear and I think that should take care of it!
Kind of overkill to use calibrate-linear for that. @williamobrien, you can double the value like this :
- name: "2"
phase_id: phase_a
input: "2"
filters:
- multiply: 2.0
Quick question - were your header holes covered/filled with some kind of gel-like/stickyish substance? I'm assuming I can just remove it/poke through it, but wanted to see if that was how they all are.
When I first opened the device, I also noticed that there was something on the PCB, so you're not alone. Most likely a dielectric gel to protect the high voltage circuitry.
Both of my vue2s also had that goop. It smelt like left over rosin. It burns off very easily with a solder iron. Also, 115200 was the highest baud rate I could set for read_flash. Any higher and I got error messages like this: A fatal error occurred: Corrupt data, expected 0x1000 bytes but received 0xf88 bytes
I am using an FTDI clone that I had for well over a decade, so I suspect this problem is on my end. Flashing went through without a hitch, but I have to hook it up this weekend to test the device for real.
I didn't have any goop at all. Do you have a picture of it? It'd be helpful in improving the instructions.
I have issues communicating with the module. I'm receiving following error: Failed to connect to Espressif device: No serial data received. Anyone has a suggestion? :)
Flip the tx and rx? Make sure you have good contact between the jumper wires and the board?
I have issues communicating with the module. I'm receiving following error: Failed to connect to Espressif device: No serial data received. Anyone has a suggestion? :)
Did you make sure to follow the steps to put the module in "download mode" using the reset and IO0 pin?
I tried few different combinations. What is exact procedure that works for you?
Flip the tx and rx? Make sure you have good contact between the jumper wires and the board?
Double check it, even tried changing it on purpose, still no luck. My assumption is that it is not in download mode. Is there an led indicating it is in right mode or any other indicator? Thanks
Flip the tx and rx? Make sure you have good contact between the jumper wires and the board?
Double check it, even tried changing it on purpose, still no luck. My assumption is that it is not in download mode. Is there an led indicating it is in right mode or any other indicator? Thanks
The way I do it is by putting a jumper wire on both the IO0 and EN (which is the reset pin) with a male end, then you put both the EN and IO0 jumper on the ESP32 shield, inorder to ground both pins. Then, remove the EN jumper from the ESP shield while keeping the IO0 jumper on the shield for at least 1-2 seconds before removing it.
You can validate that everything works correctly by opening a serial console while doing this with a baud rate of 115200 and then proceeding with the procedure as described above. If all goes well, it should output in the serial console that it's in download mode. Checking the serial log will also help you confirm that your TX/RX pins are in the right order.
I'm not sure if this is a test mode thing, but on my system I get a long beep from the buzzer if it is in "execute" mode. No beep indicates that it is in programming mode.
On Fri, Dec 24, 2021, 15:31 Care4Home @.***> wrote:
Flip the tx and rx? Make sure you have good contact between the jumper wires and the board?
Double check it, even tried changing it on purpose, still no luck. My assumption is that it is not in download mode. Is there an led indicating it is in right mode or any other indicator? Thanks
— Reply to this email directly, view it on GitHub https://github.com/flaviut/emporia-vue2-reversing/issues/1#issuecomment-1000927306, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHY2HPDGSTFQSPHRTENBVDUSTKDPANCNFSM5IRAJONA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID: @.***>
I didn't have any goop at all. Do you have a picture of it? It'd be helpful in improving the instructions.
yes, here's a picture of the goop on the header pins:
I've updated the guide to address what I think is flux on the pins, and to use the remove-freertos-task
branch, which I've found to be more stable in my use over the past week. The new revision of the guide is still here.
If you would like to see/use the old version, please take a look here.
I'd like to reach out to the interested communities[1][2][3] about this in a couple weeks or so, so I'd really appreciate criticism & compliments on the guide. Or to be told to hold off on this for a while longer :)
A video would be great. I haven’t done this yet but the instructions seem clear until I get to the remove any jumper from the usb. Does that mean remove all jumpers or just randomly pick one?
Mine looks like this , with a jumper between either VCC & 5V, or VCC & 3V3.
It ought to look like this, with no jumper attached to VCC:
Ok that makes sense. I haven’t purchased a usb to serial adapter yet. I wanted to make sure I could understand it all before spending money to dive into it.
Exciting little project... I received my Vue last week and followed flaviut's "Reverse-engineering" guide to get it talking to mosquitto / node-red running on a Raspberry Pi via MQTT. The parse_mqtt_dbg.py is pretty straight forward and outlines how to process the MQTT messages into an InfluxDB, which I am looking to reimplement in node-red. I notice that it's over 4 months old. Is there any newer code / documentation available which outlines the processing of the MQTT messages? Or is that pretty much all there is to it???
The circuit breaker on my hot tub trips a couple times each winter and I almost lost the hot tub last year due to the breaker tripping (it was well below freezing and the power was off for a few days). By monitoring the power usage I can get an alert as soon as the breaker trips again... Thanks for your hard work and dedication to this project. It really opens up this sweet little device as a powerful home automation tool.
The parse_mqtt_dbg.py is pretty straight forward and outlines how to process the MQTT messages into an InfluxDB, which I am looking to reimplement in node-red. I notice that it's over 4 months old. Is there any newer code / documentation available which outlines the processing of the MQTT messages?
That's all there is to it. process_message()
returns a Vue2Reading
, which you can use however you want. Only a few of the lines are MQTT-specific.
The reason there hasn't been any more work on that is that running fully open-source firmware is less janky (doesn't require specific IP addresses, working with copywrited firmware, or any limits on the wifi password). So all effort has gone into the FOSS firmware, through ESPHome.
The circuit breaker on my hot tub trips a couple times each winter and I almost lost the hot tub last year due to the breaker tripping (it was well below freezing and the power was off for a few days).
Wow! That's an awesome use, and I'm glad your hot tub survived!
Maybe you could use another esp32, duct tape, and one of these to make yourself a self resetting breaker :grin:
lol... I'll see what I can do about the automated breaker resetter...
Thanks again... You guys rock!
I've finally taken the plunge and ordered mine yesterday after finding this thread. Just wanted to say thanks for all the hard work, brilliant guide and give a quick bit of feedback on how things went for me. The unit and PCB I got look exactly like the pictures in the guide, I had the extra flux on the solder pads which was fairly easy to remove. My solder pads were full so I drilled them out with a tiny Dremel drill and soldered in headers. Everything from thereon was surprisingly straightforward. I've connected the cables from the unit's supply tail into a 13amp plug (black - Live, all other = Neutral), given that my consumer unit doesn't have enough spare room or a free breaker I'm going to mount the Emporia next to the consumer unit.
Backup from Ubuntu with a basic USB adapter: esptool.py version esptool.py v2.8 esptool.py -b 921600 read_flash 0 0x800000 flash_contents.bin esptool.py v2.8 Found 1 serial ports Serial port /dev/ttyUSB0 Connecting...... Detecting chip type... ESP32 Chip is ESP32D0WDQ5 (revision 1) Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None Crystal is 40MHz MAC: 40:f5:20:88:03:d8 Uploading stub... Running stub... Stub running... Changing baud rate to 921600 Changed. 8388608 (100 %) Read 8388608 bytes at 0x0 in 153.4 seconds (437.5 kbit/s)... Hard resetting via RTS pin...
Over to HA, create a new ESP device, copy over the example code from the "Setting up" guide and compile to download.
Attach USB adapter under MacOS, connect device and flash the .bin with ESPHome Flasher.
Quick look into ESPHome on HA and the device has connected ok, follow the usual "new device found" dialog and all the sensors come through.
I've hooked up one of the big CT clamps (that's all we need here in the UK for a normal domestic supply) and one of the little CT clamps on channel one. Readings come through ok and look correct, so far everything seems stable and I'm going to watch it for a bit before I take a deep breath over the weekend and try to squeeze all those 16 CT clamps into my consumer unit.
It looks me a while to get mine installed, mostly because I was daunted by the task of opening the load center. Once I started, everything went fine. Thank you for the work you guys have done. I directly uploaded the yaml with @krconv's branch, and it's working smoothly.
One thing I noticed which might help someone else to troubleshoot: If your power values see-saw between positive and negative, you have mapped the circuit to the wrong phase. I've attached a screenshot of my graph, showing before and after I fixed it to illustrate.
@flaviut's instructions are very clear on how to avoid this in the first place. My excuse is that I have a cluttered load center and somewhere along the way, I started counting wrong :)
Glad you got things working :)
I want to point out one thing: it's not unusual to get ±1W of noise in the readings, that's normal and expected. The problem your image shows is -0.25 watts (on average). Unless you have a solar system or something, that's a problem:
Just stopped by to say I have converted all four of mine and they are running stable. 1 in the main panel monitoring my mains , my solar, and my dual phase circuits. 2 in the inside panel monitoring the line from the main panel, and 28 different single phase breakers 1 in my pool subpanel monitoring the pumps, controller, lighting, and UV lights.
These CT's can be finicky and I recommend calibrating them properly. The default settings were not accurate for me and depending how how close the CT is to other wires and the panel, I see a lot of variances that need to be filtered out. I chose a median filter, sliding window of 5, with a report rate of 1. I also added a lambda filter as it bounces below 0 quite a bit (a sign that you don't have it 100% calibrated) and I just want to filter those out.
The funny part is I am using emulated kasa to feed some of these circuits to my sense, just because I like their UI better.
These CT's can be finicky and I recommend calibrating them properly. The default settings were not accurate for me
About how wrong was the data for you? I'd expect up to about 5% difference because we don't set a certain calibration constant dynamically at this time. I'm curious if this was the problem.
I chose a median filter, sliding window of 5, with a report rate of 1.
Do you find this works better than a sliding average like in the example? I've ended up with this in my personal config, but if there's something better, I'd be happy to change:
- &df1
sliding_window_moving_average:
window_size: 12
send_every: 6
- &df2
lambda: 'return max(x, 0.0f);'
and
accuracy_decimals: 0
(a sign that you don't have it 100% calibrated)
I disagree. Every system has some degree of noise to it, nothing that exists is perfect. On a circuit that consumes 0W, I'd expect to see 50% of the readings below 0W, and 50% above. Are you seeing more than ±1W of noise around 0W?
For reference, here's what I see with the filter above, and I think it's reasonable:
The funny part is I am using emulated kasa to feed some of these circuits to my sense, just because I like their UI better.
That's really cool.
That's the kind of stuff I love about hacking home automation stuff. You don't need to deal with every vendor's walled garden :)
These CT's can be finicky and I recommend calibrating them properly. The default settings were not accurate for me
About how wrong was the data for you? I'd expect up to about 5% difference because we don't set a certain calibration constant dynamically at this time. I'm curious if this was the problem.
I am seeing about 5% variance now, but originally all of my numbers were trending low when compared to other manufacturer devices monitoring the same line.
When I converted these units they had been running for a while, and the calibration numbers I was seeing in the logs/NVS were higher than the defaults. I believe in both cases I was seeing different values from defaults in the source ( 0.023109 for phase A and a different number for phase B that I forgot to write down). I am assuming this was fine tuned by emporia.. I also have other CT's monitoring these same lines (The Sense, CT's in my powerwalls, my EV wall controller is also capturing them). I slowly adjusted the values until the vue started reporting similar numbers for voltage and wattage.
I chose a median filter, sliding window of 5, with a report rate of 1.
Do you find this works better than a sliding average like in the example? I've ended up with this in my personal config, but if there's something better, I'd be happy to change:
I think it comes out to a wash in the end. I am running with your filters right now (changed the windows to still report once per second) and I don't know if I would be able to tell the difference. /shrug
- &df1 sliding_window_moving_average: window_size: 12 send_every: 6 - &df2 lambda: 'return max(x, 0.0f);'
and
accuracy_decimals: 0
(a sign that you don't have it 100% calibrated)
I disagree. Every system has some degree of noise to it, nothing that exists is perfect. On a circuit that consumes 0W, I'd expect to see 50% of the readings below 0W, and 50% above. Are you seeing more than ±1W of noise around 0W?
In some of my lines I see it jump ±20W when at 0. When I first installed my sense, their support contacted me and let me know that if the CT was pressed against the side of my panel (rigid wiring) I would see behavior like this. I am assuming my issue is similar there for these. I am going to have to open up my panels and see if I can find a more convenient spot.
The stock settings gave me low numbers originally. I think it was showing my voltage around 10V lower than it actually was, which I think was driving all of the power numbers down. Once I had that calibrated, I am seeing something similar to you in that a 0W reading bounces around ±1W.
For reference, here's what I see with the filter above, and I think it's reasonable:
The funny part is I am using emulated kasa to feed some of these circuits to my sense, just because I like their UI better.
That's really cool.
That's the kind of stuff I love about hacking home automation stuff. You don't need to deal with every vendor's walled garden :)
I am loving this as well! Once we have arduino framework support, I am looking forward to having the device perform the kasa emulation itself. :)
One more data point about the inaccuracy of the readings - one of my circuits was showing abnormal fluctuations (+/- 30W with no load) after I bottled everything up. I went back and opened the panel and I saw that the 2.5mm jack had slightly dislodged from the port because of all the wires that are crammed in there. I reseated the connection, and everything went back to normal.
Obviously, this isn't a code issue, but for someone troubleshooting inaccurate readings, this is another troubleshooting step to consider.
One more data point about the inaccuracy of the readings - one of my circuits was showing abnormal fluctuations (+/- 30W with no load) after I bottled everything up. I went back and opened the panel and I saw that the 2.5mm jack had slightly dislodged from the port because of all the wires that are crammed in there. I reseated the connection, and everything went back to normal.
That's right, I remember now! I had this exact same issue, and actually put off putting the panel cover back on (since it would move the stuff inside around) because of it.
Thanks for bringing this up, I'll add it right to the FAQ.
Thanks for all the hard work here. I flashed stock Micropython to the esp32 and can read the CTs over i2c. Here in the UK, typically we have only a single phase. I was wondering if I can read the power factor of the circuits, and report kWh as well as kVA(h).
I was wondering if I can read the power factor of the circuits, and report kWh as well as kVA(h).
As far as I'm aware, no. There may be some configuration options for the other MCU on the board, but I've never seen them so I don't know if they exist.
Is anyone having any long term stability issues? I am using github://krconv/esphome@remove-freertos-task
for a while now and I had been very happy until earlier this week when I saw that all my circuits were reading NaN.
Looking at the logs, I see this a lot:
[E][emporia_vue:055]: Failed to read from sensor due to I2C error 3
I've tried rebooting a few times and have already reflashed the firmware. I don't see how running esphome can cause this since we are only reading data from the bus, but I am curious to see if anyone else has also seen this error.
I see that same error some of the time and it usually doesn't go away without actually disconnecting the power for about 5 seconds and plugging it back in. Since mine is two phase, both circuit breakers that are providing power need to be off at the same time.
Also, you might have more luck with github://flaviut/esphome@emporia-vue
. This fork appears to be more stable for me.
Repurposing this issue for the discussion that it started. Was originally a question about connecting, and turned into a discussion about deciding the sensor data.
In the blog post, you mentioned that one of your USB-Serial adaptors wasn't working when you tried to connect to the ESP32. I'm wondering if I'm hitting the same issue; I only have one serial adaptor to test with at the moment though.
I'm using the FT232RL FTDI Mini USB to TTL Serial Converter, and getting nothing while trying to connect:
Process I'm taking to enter flashing mode:
But that's not working; my only thought is that maybe there is something else connected with TX/RX that is interfering? And a different serial converter could sort through the noise?
Any guidance would be appreciated!
Edit by @flaviut for visibility:
There is now a ESPHome component. You can follow development at