Open krconv opened 2 years ago
krconv
commented
2 years ago Actually, I had TX and RX backwards; TX on the debug header goes to RX on the ESP32. My bad (you mentioned it in the blog post and I missed it). Just successfully downloaded the flash and got debug logs
flaviut
commented
2 years ago Ha! This stuff happens, glad you got it figured out! Please feel free to ask anything else, and let me know if you end up building something on top of all this!
krconv
commented
2 years ago My main goal is to build an esphome integration. I've never had to fully reverse a chip before, but it feel like a fun challenge that I might be able to help with, and I like the small form-factor and low price of the Vue. I'd appreciate any pointers if you have any; currently waiting for a USB oscilloscope to help with identifying what each of the pins are doing.
flaviut
commented
2 years ago Sure! First off, oscilloscopes are very helpful for identifying the function of things, but when it comes to actually reading digital data you want a logic analyzer.
Anyway, please do poke at it, it's a lot of fun. If you'd like though, I briefly described the physical communication on the board here: https://flaviutamas.com/2021/reversing-emporia-vue-2#on-board-communications
There's another chip on the board that communicates with the ESP32 over I2C. The raw I2C message gets dumped over MQTT as hex here: https://github.com/flaviut/emporia-vue2-reversing/blob/master/example_messages.txt#L2-L19. Note that right below that hex data are the output results: ~the I2C data is combined with the calibration data stored in NVS to produce the final output:~ The calibration data in NVS is nulled and not used at all.
flaviut
commented
2 years ago By the way, I've given you write access to this repo! Please feel free to do whatever you wish with that access, but let's keep Emporia-copyrighted stuff off of here to avoid causing them problems :)
Maelstrom96
commented
2 years ago Hey - I'm actually working on the exact same goal as @krconv. I'm mostly trying to make it so I can add a separate ESP32 running ESPHome that would read the messages on the i2c bus and publish that data to MQTT or through the native ESPHome API, without the need to flashing the onboard module. The reason being that I like having the data in their UI, but I also want to be able to do second triggers on the energy consumption. At first I was thinking about making a MQTT bridge to their AWS IOT mqtt endpoint, but I wasn't able to reverse engineer the certificates/auth method.
Because of this, I've been working on reverse engineering the i2c message data and I think I mostly have it figured out at this point. Here is my data spreadsheet I used to figure out / validate the parsing logic: https://docs.google.com/spreadsheets/d/1Yu98tDRhFJqqMntNak6X297SKyzQtf0_Cvv_g7qKewk/edit?usp=sharing.
The only data missing from the message is the calibration - which is only on the esp32 - and the voltage frequency.
Edit: Forgot to say, thanks @flaviut for the post and the data in the repo, it was immensely useful. Also, I updated my Vue2 firmware to the last version (Vue2-1612298550, Tue Feb 2 20:42:33 UTC 2021) and it's still outputting logs and the TEST_MODE is still there and it looks like it's still working.
flaviut
commented
2 years ago WOW! This is incredible! I'm curious about how you went about doing this, I did spend a while looking at things in a hex editor, but didn't really get anywhere.
flaviut
commented
2 years ago It took me a bit to understand the spreadsheet, but now that I do, I've written up a C declaration for that data:
// all data is little-endian
struct __attribute__((__packed__)) ch_power {
int32_t ph1, ph2, ph3;
}
struct __attribute__((__packed__)) vue2_i2c_data {
uint32_t unknown1;
ch_power power[19];
int16_t ph_voltage[3];
int16_t frequency;
int16_t ph2_offset_deg;
int16_t ph3_offset_deg;
int16_t current[19];
uint16_t unknown2;
}One thing that I suspect is going on here (from my use of the Vue 2 locally) is a non-1:1 mapping of channel numbers to data indexes. I think, but am not sure, that some of the indexes here do not match up with the numbers printed on the box.
Maelstrom96
commented
2 years ago I'll start off by saying that I'm mostly out of my league when it comes down to reverse engineering things, but I wanted to give it a shot anyway.
The first thing I did was to manually check the debug/v2 message that you had in your post to see if I could find any pattern. An obvious one was the amount of FFFF value bytes separated by 2 bytes of data. I also noticed that all the P[V] values were small-ish negative ones. Doing a find and count for FFFF in Notepad++ returned 57 match. Dividing that value with the 3 P[V] per input results in 19 - the same amount of input we have in the logs. That block was then clearly the data for all the input power values.
I copied the first 12 bytes (which should correspond to the first 3 P[V] of I01) in this amazing hex tool and started to check for other patterns in the data. I also copied another 12 bytes from another message with positive values, and the values seemed to correlate when parsed as INT32 (DCBA).
I wasn't sure what to do then, since the decimal values weren't the same as the expected values. I decided to plot values from multiple inputs on a graph to see if there was a linear equation hidden somewhere.
Here's what it gave me :
As you can see, it does look like it's linear, but calculating the actual expression gave me a pretty significant offset (which can also be seen on the graph since the line is not going through 0). A small offset is expected since the values are most likely rounded, but the result is too much. Also, by looking at the graph, we can see that there are 2 general groups. I was able to see that the slope value was different between I01-I03 and I04+.
After thinking about it, and checking the points (expected value and RAW decimal) on the graph, I noticed that not all the points were exactly on the trend line, so I started to wonder if the calibration numbers could be at play, since it would explain the deviation. I decided to multiply the values by their respective calibration and plot those values.
Here's the graph after that :
As you can see, the deviation became negligible, and the offset was almost 0 (still some but very small, > 0.03). This told me I should now be able to calculate the slope and then use that to get the P[V] value. The slope 5.5% for the first 3 inputs, and 22% for the rest.
After that, I was wondering what other values I could find, and decided to enter the next couple of bytes into the HEX tool, and got three values that I could spot easily in the debug output:
pos | Raw | INT16 -- | -- | -- 0 | 00 0F | **15** 2 | 00 0C | **12** 4 | 00 0B | **11**V1: 0.3, 0.4 Hz, *15*, 0.0225336
V2: 0.3, 0 degrees, *12*, 0.0220000
V3: 0.2, 0 degrees, *11*, 0.0220000I wasn't sure what those values where, but I decided to see if multiplying this with the calibration data would actually give the voltage, and it actually did. So I figure this is a voltage ADC value.
At this point, the amount of bytes of data was pretty low, not enough for 19 x int32, I also noticed that the voltage ADC was an int16, so I started to see if I could notice any pattern with 19 x int16 in a row. It was pretty easy to spot since all the values were pretty much the same, see for yourself :
0050 0050 0050 FD4F FD4F FD4F FD4F FD4F 0350 0350 0350 FD4F FD4F FD4F FD4F FD4F 0350 0350 0350
I01: 372.4, P[V1]: -80.0, P[V2]: -21.9, P[V3]: -4.9
I02: 372.4, P[V1]: -80.0, P[V2]: -21.9, P[V3]: -4.9
I03: 372.4, P[V1]: -80.0, P[V2]: -21.9, P[V3]: -4.9
I04: 93.1, P[V1]: -19.4, P[V2]: -5.2, P[V3]: -0.9
I05: 93.1, P[V1]: -20.3, P[V2]: -5.6, P[V3]: -1.7
I06: 93.1, P[V1]: -20.5, P[V2]: -5.8, P[V3]: -1.3
I07: 93.1, P[V1]: -19.2, P[V2]: -4.4, P[V3]: -2.4
I08: 93.1, P[V1]: -20.0, P[V2]: -5.7, P[V3]: -0.7
I09: 93.1, P[V1]: -20.9, P[V2]: -4.8, P[V3]: -1.0
I10: 93.1, P[V1]: -19.4, P[V2]: -6.2, P[V3]: -1.0
I11: 93.1, P[V1]: -20.1, P[V2]: -6.1, P[V3]: -0.8
I12: 93.1, P[V1]: -19.4, P[V2]: -5.2, P[V3]: -0.9
I13: 93.1, P[V1]: -20.3, P[V2]: -5.6, P[V3]: -1.7
I14: 93.1, P[V1]: -20.5, P[V2]: -5.8, P[V3]: -1.3
I15: 93.1, P[V1]: -19.2, P[V2]: -4.4, P[V3]: -2.4
I16: 93.1, P[V1]: -20.0, P[V2]: -5.7, P[V3]: -0.7
I17: 93.1, P[V1]: -20.9, P[V2]: -4.8, P[V3]: -1.0
I18: 93.1, P[V1]: -19.4, P[V2]: -6.2, P[V3]: -1.0
I19: 93.1, P[V1]: -20.1, P[V2]: -6.1, P[V3]: -0.8I wasn't sure why there were 2 different values FD4F & 0350 for what looked like 93.1, but I was still pretty sure that they were the right bytes. I tried to multiply one of the values with it's input offset annnnnnnnnnnd:
0x0050 = 20480
20480 x 5.5 = 112,640Well, that doesn't look anything close to 372.4... How about a division?
20480 / 5.5 = 3,723.63636363636[...]That was mostly pure luck to find that, but it looks like it is indeed how it's calculated. Just need to shift the period and we're golden!
I know it's pretty useless and not needed, but at this point I feel like superman, so I wanted to see If I could crack the code.
The amount of bytes left is pretty limited : 03EC 52EA [...] A601 8E00 0000 [...] 0000
There's only a single block with 3 int16, so it's only logical that A601 8E00 0000 are the right ones, but converting them didn't really give useful numbers :
V1: 120.2, 61.6 Hz, 5241 ADC, 0.0229308 Calibration
V2: 121.3, 121 degrees, 5574 ADC, 0.0217630 Calibration
V3: 8.1, 0 degrees, 369 ADC, 0.0220000 CalibrationThe 0 degree made sense, but how about the 142? How does that become 121 degrees? I wondered if the values were a ratio of 360°. Calculating it actually gives the right value.
142 x
--- = ----
422 360
x = 360 x 142 / 422
x = 121.13[...]I haven't figured out if the voltage frequency is in the data, but I don't think it is, since the first 4 bytes seems to be some kind of counter, and the last 2 bytes are always 0x0000. It could actually be the 422 value, but I would need to have more data with variations to confirm it.
Maelstrom96
commented
2 years ago One thing that I suspect is going on here (from my use of the Vue 2 locally) is a non-1:1 mapping of channel numbers to data indexes. I think, but am not sure, that some of the indexes here do not match up with the numbers printed on the box.
If you have a message that you think doesn't match the mapping, let me know. I don't believe that it will deviate from this structure but we never know.
Maelstrom96
commented
2 years ago @flaviut I wanted to know - do you happen to know the i2c frequency? I'm trying to connect to it but I'm not having much luck at 50kHz or 100kHz.
flaviut
commented
2 years ago @Maelstrom96 I believe it's 100kHz. I uploaded a dump from sigrok pulseview to https://github.com/flaviut/emporia-vue2-reversing/blob/master/i2c%20dump.bin, and unless the time base is off in that, it shows 100kHz.
That hopefully the screenshot will also explain how it expects to be asked for data.
krconv
commented
2 years ago This is awesome! I just had fun decoding the message too, and it's cool to see that I landed in the same place. I used Python to help me walk through the message, activating each of the inputs and finding the bytes that changed. In case your curious, this is my helper script. I was just about do try to figure out the calibration as well, and ended up with a linear relationship (just eye-balled it, didn't verify it was exactly linear) between points on I04[V2], similar to you.
From my testing, I'm pretty sure the uint16 with offset 0x00EE is the frequency; it was sandwiched between the AC voltages and degrees, and it was consistently changing with the frequency output in MQTT. The conversion for my device is 0.0014268004584473604 * <sensor_value> + 0.4 = frequency; tried it on the example messages @flaviut and it was a little off, but I think that is due to a different calibration.
For calibration data; I think we could do the same thing that the existing CT clamp sensor in ESPhome does, where whoever is flashing the device would need to manually calibrate each of the sensors. Maybe we could reverse engineer the calibration data from nvs, and make that easier? I think it's worth skipping that for now.
@Maelstrom96 Would you want to work together on the integration? I'm not familiar with building an ESPHome integration, are you? I am familiar with C++ and think I could make faster progress on the integration though if you want to keep looking into the connections on the board because I'm not as familiar with the signal processing side of things.
krconv
commented
2 years ago And I'm pretty excited...I feel like this will be a huge improvement for cost effective energy monitoring if we can get it to work
krconv
commented
2 years ago Reopening for visibility, to make it easier to find for others interested
Maelstrom96
commented
2 years ago @krconv I'm not really familiar with ESPHome components. I tried to see how ESPHome handles I2C devices and I'm not sure if we're going to have an issue with the TwoWire lib that they're using. The reason being that is looks like requests read buffer is limited to 32 bytes with TwoWire, and we need to query the full 284 bytes all at once.
krconv
commented
2 years ago I've been trying to learn how various ESPHome components work. I think under the hoods, i2c will use the TwoWire library by default, but we can configure it to talk to the onboard i2c controller instead and I'm guessing that's how the the original software does it.
I'm not sure what other implications this has, but this is an example from here to tell ESPHome to use the native ESP API:
esp32:
board: esp32-c3-devkitm-1
framework:
type: esp-idf
version: recommendedWe'll most likely have an issue with https://github.com/espressif/arduino-esp32/blob/master/libraries/Wire/src/Wire.h too. And if you look at Wire.cpp, the requestFrom() implementation cast down the "len" value to an uint_8, which doesn't go high enough for our 284 bytes.
krconv
commented
2 years ago I tried out that approach, and agreed I don't think it's much of an option. I think setting the framework type to esp-idf will cause it to load this driver (instead of any of the Arduino ones), and from what I can tell it doesn't have any internal buffers, which I think is good. But, setting the framework type also causes everything to attempt to compile using the native framework, and there isn't one for MQTT according to esphome, so validation throws this error:
From what I understand, Arduino is a generic framework, and arduino-esp32 are the bindings written to port Arduino to ESP32. ESPHome just barely added support for using the native esp-idf framework in release 2021.10. It seems like if we could get @flaviut's PR merged to the esp8266 and esp32 Arduino libraries, we could keep doing I2C through Arduino, or if we have enough support on esp-idf, we could go in that direction (because this component would only need to be used on esp32 anyways).
Also, I've been thinking about a potential Config format; have either of you tried that out yet? Any thoughts on this idea?
sensor:
- platform: emporia_vue
update_interval: 2s
phases:
- id: phase_a
phase_input: BLACK # i.e. the color of the wire that is connected to the phase
- id: phase_b
phase_input: RED
power:
- id: total_phase_a
phase_id: phase_a # not sure how the factory software figures out which phase to use; this makes it explicit
ct_input: A # uses A-C, 1-16 as labeled on the outside of the Vue
filters: # and each one would have a separate calibration
- calibrate_linear:
- 0 -> 0
- 2393 -> 2.0
- id: kitchen_outlets_power
phase_id: phase_a
ct_input: "1"
filters:
- calibrate_linear:
- 0 -> 0
- 2393 -> 2.0
- id: stove_power
name: Stove Power
phase_id: phase_b
ct_input: "2"
filters:
- calibrate_linear:
- 0 -> 0
- 3438 -> 2.0
- lambda: return x * 2;
I've been testing out config validation on my fork using this external component:
external_components:
- source: github://krconv/esphome@add-emporia-vue
components: [ emporia_vue ]
refresh: 1minI haven't tested anything on hardware yet, but my next step might be to connect a spare ESP32's i2c port to the Vue and see if it can talk to the sensor at all with the esp-idf library
flaviut
commented
2 years ago cal_data is a false lead--the calibration data is internal to the ESP32 for their wifi hardware.
The real calibration constants are cReal and cApparent.
However, the values here are surprising. The interesting data is marked as erased, while the uninteresting version is not:
{
"entry_state": "Erased",
"entry_key": "cReal",
"entry_data": "AAAgKJkOI0EAAOC+axEjQQAA8OEE2SFBAACAbhjnAkEAAHCC3dECQQAAhAty2wJBAADg+pYAA0EAAKhAYtoCQQAA2O8s5wJBAAAkLELwAkEAADA5JMoBQQAAtLRw6wJBAAC4VIYKA0EAACjF59kCQQAAYHSiBwNBAAAwksz1AkEAALj8Wt8CQQAA6OxK6AJBAAAQYf0SA0E="
},
{
"entry_state": "Erased",
"entry_key": "cApparent",
"entry_data": "AACA/dMMwEAAAACbBxDAQAAAgB+DMcFAAAAAb/v/n0AAAACo+dufQAAAABPd7p9AAACAwU0VoEAAAACnb+ifQAAAAAaV/Z9AAACAftMIoEAAAIAUqTShQAAAAJyuA6BAAAAAsfAboEAAAACm7+ifQAAAgOacGaBAAAAAIswKoEAAAAB68PKfQAAAAEllAaBAAACAiX0koEA="
},
{
"entry_state": "Erased",
"entry_key": "cReal",
"entry_data": "AAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEA="
},
{
"entry_state": "Written",
"entry_key": "cApparent",
"entry_data": "AAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEA="
},Decoding these gets you basically what you expect: 19 entries, the first 3 of which are different than the rest.
$ base64 -d | xxd
AAAgKJkOI0EAAOC+axEjQQAA8OEE2SFBAACAbhjnAkEAAHCC3dECQQAAhAty2wJBAADg+pYAA0EAAKhAYtoCQQAA2O8s5wJBAAAkLELwAkEAADA5JMoBQQAAtLRw6wJBAAC4VIYKA0EAACjF59kCQQAAYHSiBwNBAAAwksz1AkEAALj8Wt8CQQAA6OxK6AJBAAAQYf0SA0E=
00000000: 0000 2028 990e 2341 0000 e0be 6b11 2341 .. (..#A....k.#A
00000010: 0000 f0e1 04d9 2141 0000 806e 18e7 0241 ......!A...n...A
00000020: 0000 7082 ddd1 0241 0000 840b 72db 0241 ..p....A....r..A
00000030: 0000 e0fa 9600 0341 0000 a840 62da 0241 .......A...@b..A
00000040: 0000 d8ef 2ce7 0241 0000 242c 42f0 0241 ....,..A..$,B..A
00000050: 0000 3039 24ca 0141 0000 b4b4 70eb 0241 ..09$..A....p..A
00000060: 0000 b854 860a 0341 0000 28c5 e7d9 0241 ...T...A..(....A
00000070: 0000 6074 a207 0341 0000 3092 ccf5 0241 ..`t...A..0....A
00000080: 0000 b8fc 5adf 0241 0000 e8ec 4ae8 0241 ....Z..A....J..A
00000090: 0000 1061 fd12 0341 ...a...A
$ base64 -d | xxd
AAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEAAAAAAAAAkQAAAAAAAACRAAAAAAAAAJEA=
00000000: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000010: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000020: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000030: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000040: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000050: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000060: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000070: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000080: 0000 0000 0000 2440 0000 0000 0000 2440 ......$@......$@
00000090: 0000 0000 0000 2440 ......$@I bet that these can be decoded as:
struct __attribute__((__packed__)) calib_entry {
int32_t offset, slope;
}
typedef calib_entry[19] calibration;@Maelstrom96 You're right--increasing the number there won't work unless we adjust the function signatures too. We already have https://github.com/espressif/arduino-esp32/blob/master/libraries/Wire/src/Wire.h#L98, and it's a real shame that that function is written in terms of the other functions rather than having all the other functions call it. However, it seems like a simple-enough refactoring, and the ESP32-arduino project seems responsive to contributions.
@krconv
framework type to esp-idf will cause it to load this driver (instead of any of the Arduino ones), and from what I can tell it doesn't have any internal buffers, which I think is good. But, setting the framework type also causes everything to attempt to compile using the native framework
:(
I guess you could try adding MQTT support to the esp-idf version of the esphome project, that'd probably be something useful for everyone. But I think the fastest & easiest way would be to do PRs to the wire library, since they're pretty simple. I can do that over the next day or two!
Also, I've been thinking about a potential Config format; have either of you tried that out yet? Any thoughts on this idea?
Huh. Neat! I had no idea ESPHome made things that easy, that's really nice!
krconv
commented
2 years ago The calibration data I think will be helpful, the more I think about it. Calibrating 19 CT clamps manually might be a real pain; maybe we could provide a script that helps parse the calibration data later on.
That sounds great; I agree getting MQTT on esp-idf would probably be helpful for everyone, but ya it might slow things down. I also checked the api component, and that relies on Arduino too; so there would be no way to get the sensor data to home assistant without some more development. I did find a native MQTT library; if we do go that route, we might be the first integration that requires esp-idf, which might have some other implications. I'll take a look at the work involved to support MQTT on esp-idf and see if the i2c using the native library even works for this scenario.
flaviut
commented
2 years ago So I did some digging, and here's how things work:
The NVS calibration on the Emporia Vue 2 that I got isn't used. I'm not sure why the cReal and cApparent NVS values exist, but they are set to zero and not used. The last bit of each entry is probably a valid bit, because it is 0 in all of the zeroed entries, and 1 in the original, non-zero entries.
However, that doesn't mean the Vue 2 has no calibration at all. After all, we saw that it does have some sort of calibration constants baked into the firmware from @Maelstrom96's message reverse engineering efforts. What it does have is the following array:
From trying to read the decompiled code, it seems like the calibration constants it chooses are based on how the phases are aligned on the input wires. I'm having a lot of trouble understanding what the code is doing, but it's around 401040dc in the ELF file, and looks something like this when decompiled:
Maelstrom96
commented
2 years ago so there would be no way to get the sensor data to home assistant without some more development.
The native ESPHome API will be able to export the data to Home Assistant without the need for MQTT. Once the updates to the arduino libs are done, everything should work on the Arduino and esp-idf using their I2CDevice class. Right now, you should be able to test it without MQTT fine.
Also, there's still 2 things that I'm not sure what to do about them :
Calibration : I wonder if the calibration is really changing that much per CT. When doing my calculations, I was applying a single calibration value on every CT per phase, and it was returning the correct values that were sent out in the debug messages. Getting the ~0.022 value per phase seems to be easy enough since it's being printed to Serial, and could be captured before flashing ESPHome on the ESP32 quite easily.
"Current" : The thing is that I doubt that the "current" number is actually instantaneous current - I'm pretty sure it's some kind of current counter that allows the ESP32 to know how much current has passed in total across the CT. It would be a good idea to try to reverse this value to something we can use to properly calculate kWh values per CT. The Power values can't generate a reliable total energy consumption value alone. The way we could do that is by applying a constant load through a CT and try to find a correlation with the "current" value.
Also, I might try it out your fork @krconv on my bench today if I get the time.
flaviut
commented
2 years ago TLDR because my message above was basically a dump of my thoughts:
I wonder if the calibration is really changing that much per CT
Calibration is not different per CT. Calibration is different depending on the phase order on the voltage sensing side in some way I do not understand.
Getting the ~0.022 value per phase seems to be easy enough since it's being printed to Serial, and could be captured before flashing ESPHome on the ESP32 quite easily.
Yup, this is probably the easiest way to do things. And there's only 6 combinations of wires, so the wire->calibration constants could be tested manually pretty easily.
flaviut
commented
2 years ago The thing is that I doubt that the "current" number is actually instantaneous current
For some reason, I thought it was spot-on. It isn't quite, looking at a recent utility bill:
But that still is very close, especially with the only calibration here being a global, per-voltage calibration that's pre-set from the factory and not device-specific.
edit: both misread my bill, and excluded a day in my report
Maelstrom96
commented
2 years ago
- I'm being billed for 873kWh
- Integrating & summing the two phases comes out to 752.2kWh
- Integrating & summing the various circuits comes out to 752.5kWh
Yeah, that would be the kind of variation I would expect if it wasn't being counted correctly - Definitely something to investigate, since many people on the Emporia forum seems to say that the totals are really close to their utility bills (which makes sense since the 200A CT should rarely be below ~1A, their minimum for accurate readings)
Maelstrom96
commented
2 years ago Also, something interesting I noticed is that in the Pulseview dump from @flaviut, the intervals between data requests on the i2c bus aren't every second like I expected. They seem to be every ~240ms.
Something else that is intriguing is that the debug messages always seems to start by 0x03, and @krconv documented it as the message "version", but I'm seeing it with a value of 0x03 and 0x00 sometimes.
The first two bytes and the two bytes after seems to be some kind of counters, and, in the dump, the first one seem to alternate every message, while the second value is updated every 2 messages. E.g. msg1-0x03A2, msg2-0x00A2, msg3-0x0306, msg4-0x0006, msg5-0x0391, msg6-0x0091.
So I think It might indicate something else and be related to the "current" value, so we can sum it correctly.
Maelstrom96
commented
2 years ago
- I'm being billed for ~873kWh~ 790kWh
- Integrating & summing the two phases comes out to ~752.2kWh~ 782.1kWh
- Integrating & summing the various circuits comes out to ~752.5kWh~ 782.2kWh
Do you have the current formula / procedure that you use to get from the current value to those power consumptions?
flaviut
commented
2 years ago I completely ignore the current. I used this code to ingest the data into InfluxDB: https://github.com/flaviut/emporia-vue2-reversing/blob/master/parse_mqtt_dbg.py#L196-L242
And this query to process it:
import "math"
from(bucket: "home_automation")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "home_power")
|> filter(fn: (r) => r["_field"] == "power")
|> filter(fn: (r) => r["phase"] != "")
|> group(columns: ["_start", "_stop", "phase"])
|> integral(unit:1h)
|> group()
|> keep(columns: ["_value", "phase"])I don't really understand why the integral() function takes a unit
flaviut
commented
2 years ago Something else that is intriguing is that the debug messages always seems to start by 0x03, and @krconv documented it as the message "version", but I'm seeing it with a value of 0x03 and 0x00 sometimes.
Looking at the code, it seems like the existing code mostly ignores messages that do not have it set to 0x03:
if ((iVar1 == 0) && (pcVar6 = pcVar2 + 0x400, pcVar2[0x410] == '\x03')) {
pcVar7 = pcVar2 + 0x410;
pcVar8 = (char *)0x11c;
pvVar16 = memcpy(&uStack1664,pcVar7,0x11c);
FUN_40104f54(pvVar16,pcVar7,0x11c,line_00,pcVar9);
FUN_40109564(auStack104,pcVar7,0x11c,line_00,pcVar9);
pcVar7 = pcVar2;
FUN_4010432c(auStack332,pcVar2,0x11c,line_00,pcVar9);
pcVar6 = pcVar2;
}
uVar5 = FUN_40092448(0,pcVar6,pcVar8,line_00,pcVar9);Of course, it's extremely likely I'm misunderstanding what is going on here.
Maelstrom96
commented
2 years ago Looking at the code, it seems like the existing code mostly ignores messages that do not have it set to
0x03:
I wouldn't be surprise, it would make sense that this would indicate to the ESP that the data is "ready". An 0x00 would indicate that the data is being updated, and then that would explain why the third and fourth bytes have a new value when we get a new 0x03.
Maelstrom96
commented
2 years ago I completely ignore the current. I used this code to ingest the data into InfluxDB
Looking at the code, you still seem to send the current into the influxDB. Would it be possible for you to dump the full timeseries for the period that match your known power usage (from your utility provider), so we could try and calculate the current formula that might give an even more precise power consumption value.
flaviut
commented
2 years ago Here's the full data for the past 30 days: https://drive.google.com/file/d/14VMxbSIGjlCfecbEYWAQfevvbe-ccUAM/view?usp=sharing
Note: it extracts to 5.4GB of data.
| Day | kWh |
|---|---|
| Oct 27 | 17 |
| Oct 28 | 19 |
| Oct 29 | 15 |
| Oct 30 | 10 |
| Oct 31 | 21 |
| Nov 01 | 22 |
| Nov 02 | 22 |
| Nov 03 | 20 |
| Nov 04 | 19 |
| Nov 05 | 22 |
| Nov 06 | 21 |
| Nov 07 | 28 |
| Nov 08 | 19 |
| Nov 09 | 21 |
| Nov 10 | 26 |
| Nov 11 | 22 |
| Nov 12 | 21 |
| Nov 13 | 27 |
| Nov 14 | 18 |
| Nov 15 | 23 |
| Nov 16 | 31 |
| Nov 17 | 18 |
| Nov 18 | 18 |
| Nov 19 | 24 |
| Nov 20 | 19 |
| Nov 21 | 21 |
| Nov 22 | 22 |
| Nov 23 | 23 |
| Nov 24 | 10 |
| Nov 25 | 11 |
| Nov 26 | 23 |
| Nov 27 | 29 |
krconv
commented
2 years ago A couple observations I've had:
total_daily_reading component, which seems to be able to take in a sensor with power information and sum it up over the dayThe thing is that I doubt that the "current" number is actually instantaneous current
What if it is actually "the current passed since last reading"? i.e. the current that's passed since the last message with a 0x3 "version" header.
Theory: The ESP polls the sensor every ~240ms. Meanwhile, the sensor is on some clock where it'll calculate "instantaneous" (or ~2 second window) current draw, voltage, and calculate power consumption. Maybe the "version" acts as a "is new reading" flag, where reading it causes it to be cleared. To calculate total power draw accurately, every single sensor reading would need to be aggregated together, because missing one measurement would mean missing the power consumption in that ~2s window.
I'm still having a hard time understanding how AC power is calculated (i.e. I understand DC P=V*I, but the integrals for AC I might not understand right). But, if the "power" sensor readings really are power readings, and have a unit of W, then wouldn't we only need to sum those up over a time period to get Wh/KWh? That is where I think the total_daily_reading component might come in, because it a) calculates the time since last reading and b) multiplies the time delta by the instantaneous power reading and c) adds it to a total counter (which I think can be consumed by home assistant's energy dashboard)
krconv
commented
2 years ago I flashed some code to my Vue, and am successfully reading from I2C on the esp-idf framework. I also tested using Arduino (just to see what would happen without the bigger buffer), and the I2C call returns a 28 error code; couldn't find what that means though.
Here's the output being spit out from ESPHome
[VV][scheduler:152]: Running interval 'update' with interval=1000 last_execution=4294966081 (now=580)
[VV][i2c.idf:137]: 0x64 RX 030452A6C8FDFFFFDDFDFFFFE7FDFFFFC7B9FAFFC7B6FDFF18BEFDFFEA050000E5020000ED020000120500009EFFFFFFABFFFFFF69070000530100006D0100003FF8FFFFF0F9FFFFFFF9FFFFF861FBFFC63AFEFFC64BFEFFDF0FFBFFF6D4FDFFE7AFFDFF6FF70A00B32708002DEF07006B400B00EB330800EF1A08004C930A000AF507000DD20700CE58FBFF7128FEFF121CFEFF6781FAFFC2A8FDFF68B6FDFF3E45FBFF2A2AFEFF8763FEFFF861FBFFC63AFEFFC64BFEFFDF0FFBFFF6D4FDFFE7AFFDFF98E4F9FFAD17FDFFBB32FDFFAD0EFAFFEB42FDFFEF1AFDFFAA49FAFF4951FDFFEF35FDFF923D37332F33A
[VV][emporia_vue:031]: Raw Sensor Data: 03.04.52.A6.C8.FD.FF.FF.DD.FD.FF.FF.E7.FD.FF.FF.C7.B9.FA.FF.C7.B6.FD.FF.18.BE.FD.FF.EA.05.00.00.E5.02.00.00.ED.02.00.00.12.05.00.00.9E.FF.FF.FF.AB.FF.FF.FF.69.07.00.00.53.01.00.00.6D.01.00.00.3F.F8.FF.FF.F0.F9.FF.FF.FF.F9.FF.FF.F8.61.FB.FF.C6.3A.FE.FF.C6.4B.FE.FF.DF.0F.FB.FF.F6.D4.FD.FF.E7.AF.FD.FF.6F.F7.0A.00.B3.27.08.00.2D.EF.07.00.6B.40.0B.00.EB.33.08.00.EF.1A.08.00.4C.93.0A.00.0A.F5.07.00.0D.D2.07.00.CE.58.FB.FF.71.28.FE.FF.12.1C.FE.FF.67.81.FA.FF.C2.A8.FD
[V][sensor:062]: 'A': Received new state -0.592032
[D][sensor:113]: 'A': Sending state -0.59203 W with 0 decimals of accuracy
[V][sensor:062]: 'C': Received new state 1.578056
[D][sensor:113]: 'C': Sending state 1.57806 W with 0 decimals of accuracy
[V][sensor:062]: '1': Received new state 1.352917
[D][sensor:113]: '1': Sending state 1.35292 W with 0 decimals of accuracy
[V][sensor:062]: '2': Received new state 1.977260
[D][sensor:113]: '2': Sending state 1.97726 W with 0 decimals of accuracy
[V][sensor:062]: '3': Received new state -2.068984
[D][sensor:113]: '3': Sending state -2.06898 W with 0 decimals of accuracy
[V][component:186]: Component emporia_vue.sensor took a long time for an operation (0.18 s).
[V][component:187]: Components should block for at most 20-30ms.```It seems like it's parsing it correctly according to the struct definition, and can successfully get the data from I2C. I compared the power values for a known load with ESPHome installed vs flashing the original software on there, and the values seemed close. I noticed with ESPHome, the values fluctuate between -2 W and 2 W though, where it is exactly zero with the original software; I wonder if the original software filters out readings below 2 watts or something like that.
This is the config I'm testing with:
substitutions:
device_name: home-power-monitor
friendly_name: Home Power Monitor
assigned_ip_address: "192.168.86.223"
esphome:
name: ${device_name}
esp32:
board: esp-wrover-kit
framework:
type: esp-idf
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
use_address: ${assigned_ip_address}
ota:
logger:
level: VERY_VERBOSE
i2c:
sensor:
- platform: emporia_vue
update_interval: 1s
phases:
- id: phase_a
input: BLACK
power:
- name: "A"
phase_id: phase_a
input: "A"
- name: "C"
phase_id: phase_a
input: "C"
- name: "1"
phase_id: phase_a
input: "1"
- name: "2"
phase_id: phase_a
input: "2"
- name: "3"
phase_id: phase_a
input: "3"
external_components:
- source: github://krconv/esphome@add-emporia-vue
components: [emporia_vue]
refresh: 1min
Maelstrom96
commented
2 years ago Wow, good job on the code!
Here are some things I found while looking at the code and log output :
The update loop seems to take ~180ms, which is way too much. But looking at the default I2C baud rate (which I believe is what's being used by your component), it's 50kHz, which is less than the default 100kHz that the Emporia firmware was using. 284 bytes at 50kHz takes around 70ms to read, vs ~35ms @ 100kHz. We should override the i2c default to at least 100kHz. Per this doc, it looks like we should be able to get away with even faster baud rates, like 400kHz (~9ms), shaving precious execution time on i2c_device::read(). Keep in mind that the baud rate is set by the master (the ESP32 in this case) so the slave should follow whatever rate we set, if it can process the signal at this rate.
Because of the way the interaction with the co-MCU works, I don't think we should tie it to the sensor refresh update() loop. We should create a separate sub-task/routine to allow precise timings on the i2c reads (maybe similar to the Emporia firmware with it's 240ms pull interval), allowing us to properly collect the co-MCU data and calculate the power usage. The update() routine would then just request the most up-to-date data or the accurate cumulative energy values from that subroutine. Also, when tying the request to the update() loop, we risk hitting a message that starts with 0x00, which we should ignore.
Your calibrated power doesn't seem to factor in the difference in factor between CT 0-2 (5.5) and CT 3-18 (22) when checking this line : https://github.com/krconv/esphome/blob/bcce62b67517046ad2c2f800d17d0c817a78be43/esphome/components/emporia_vue/emporia_vue.cpp#L60
Your calibrated power also seems to use a single calibration value. We should have an array of 3 calibration values (1 per phase) and apply the calibration value of the phase the CT is configured with.
I'm not sure if the console was overflowing, but both your serial print from i2c.idf and emporia_vue are missing some bytes. It sure does like it, since they both stopped at the same character count, and at different hex bytes.
Edit: Here's a good example on how we could start a subtask on a separate thread that would handle the i2c reads and power value calculations in a non loop-blocking way : https://github.com/esphome/esphome/blob/939fb313df622d3fabc14ea837e296c7aa5ec0ec/esphome/components/esp32_camera/esp32_camera.cpp#L35
Maelstrom96
commented
2 years ago I also tested using Arduino (just to see what would happen without the bigger buffer), and the I2C call returns a 28 error code
@krconv I'm 99% sure that it's this here : https://github.com/esphome/esphome/blob/939fb313df622d3fabc14ea837e296c7aa5ec0ec/esphome/components/i2c/i2c_bus_arduino.cpp#L75
It's just telling you that the amount of byte read is not equal to the requested len, which would be accurate since we're overflowing the uint8_t.
Maelstrom96
commented
2 years ago @krconv Would it be possible for you to give us write access to your esphome fork so we can all work on the same repo?
flaviut
commented
2 years ago This is very exciting! Nice job @krconv, I've also read all your code & it looks very clean!
With regards to the calibration, I thought you were planning to use ESPHome's built-in linear calibration? Is this change a user-friendliness change, or does using the built-in thing break something else?
krconv
commented
2 years ago Thanks for the review on the code; really helpful, I'm still trying to understand how to work with ESPHome. Just gave you both access, feel free to push directly or PR as you desire. The calibration code I had in there was mainly for local testing to see if I could get anything consistent with real values (I was uploading to GitHub and downloading it onto my Home Assistant machine to test); I added a few more TODOs to mark things that aren't done yet too.
I tried out 400kHz, which seems to work sometimes, but also sometimes times out ([20:59:43][VV][i2c.idf:119]: RX from 64 failed: timeout). Tried 200kHz, which seems to be reliable, and tried out 800kHz to see what would happen, and that causes I2C timeouts on every read.
For calibration; I'm still wrapping my head around it, and I don't feel strongly either way. I think using some "fake" calibration data to at least get the numbers to seem more understandable might be good for user friendly-ness, even if they aren't accurate, and could include instructions for how to calibrate further. If getting the phase calibrations from logs works as we expect, I think we should also give an option to plug those in directly as well to help avoid manual calibration. I still don't trust the readings, and I kinda think we're gonna run into something that encourages us to use the other sensor data that we aren't currently reading (maybe the counter in the last byte of the header or the current data); I feel like there's so much going on in the logs, and other pins transmitting data on the board that reading the sensor data directly feels like we're cheating somehow.
I like @Maelstrom96 idea to control the how often we poll from the MCU ourselves, and let how often we actually publish the value be controlled by config. Tonight I tested calibrating one of my circuits to read accurate values (using calibrate_linear), and then adjusting how often we read from the sensor to see how it affects the readings. It seems like when we read from the sensor, we only get the last ~1s of data, so if we wait too long then we miss data. For example, I set the update interval to 5s, and turned the stove on for 4 seconds, and turned it off right before the reading, and the sensor just returned a reading of 0W, instead of an average over the 5 seconds. So it seems like we'll need to implement that averaging mechanism.
It is really cool to see how fast the values come into Home Assistant. It's realtime, instead of the ~5 second delay with the Emporia app. I get a kick out of turning the stove on and off just to watch the numbers bounce back and forth 😆
Maelstrom96
commented
2 years ago It seems like when we read from the sensor, we only get the last ~1s of data, so if we wait too long then we miss data.
So my latest theory is that the co-MCU is averaging the last ~480ms of power consumption into the power readings values. So in order to get accurate total energy counts, we might need to compile the values using all the message data as soon as it's updated.
Maelstrom96
commented
2 years ago I don't have the time to test it right now, but here are some changes I made to the codebase if you want to test them @krconv : https://github.com/Maelstrom96/esphome/tree/add-emporia-vue2
Edit: Just thought about it and you might not be able to test it using the "External_Component" configuration since I made changes to the esphome/const.py file.
Edit2: Changed the code so it could be correctly imported for testing. Also, you will have to rename the "power" configuration to "ct".
krconv
commented
2 years ago I can take a look tonight! I've been playing around with putting the read task in the loop(), interested to see how the scheduled task works. BTW, I noticed that averaging a few reading together makes the readings around 0W more stable which is encouraging.
Got my the two 400A clamps on the main power cables, and running that for a day to see if our calculations are close to my power company's readings 🤞
Maelstrom96
commented
2 years ago @krconv Not sure if it's just something with my setup, but when using the esp-idf framework instead of the default Arduino one, the wifi just doesn't want to connect, and the serial/UART output is all over the place. I've tried installing the code without the emporia_vue module and by cleaning the build files to make sure that wasn't playing a role, and it's still happening with a pretty bare configuration file. Are you also seeing this? I'm using the latest ESPHome 2021.11.04.
[V][wifi_esp32:434]: tcpip_adapter_dhcpc_stop failed: ESP_ERR_ESP_NETIF_DHCP_ALREADY_STOPPED
[E][wifi:291]: wifi_sta_connect_ failed!
[D][wifi:370]: Starting scan...
[V][wifi_esp32:647]: Event: WiFi Scan Done status=0 number=18 scan_id=177
[D][wifi:385]: Found networks:
[I][wifi:429]: - 'AJ - IOT' (FA:92:BF:5D:EF:A6) ▂▄▆█
[D][wifi:430]: Channel: 5
[D][wifi:431]: RSSI: -38 dB
[I][wifi:429]: - 'AJ - IOT' (FA:92:BF:5D:F4:37) ▂▄▆█
[D][wifi:430]: Channel: 5
[D][wifi:431]: RSSI: -52 dB
[D][wifi:434]: - 'AJ - 2.4Ghz' (FE:92:BF:5D:EF:A6) ▂▄▆█
[D][wifi:434]: - 'AJ' (F4:92:BF:5D:EF:A6) ▂▄▆█
[D][wifi:434]: - '' (02:92:BF:5D:EF:A6) ▂▄▆█
[D][wifi:434]: - 'AJ' (F4:92:BF:5D:F4:37) ▂▄▆█
[D][wifi:434]: - 'AJ - 2.4Ghz' (FE:92:BF:5D:F4:37) ▂▄▆█
[D][wifi:434]: - '' (02:92:BF:5D:F4:37) ▂▄▆█
[D][wifi:434]: - 'JLWIFI' (90:AA:C3:37:0F:58) ▂▄▆█
[D][wifi:434]: - 'yeelink-light-color2_miapC64C' (04:CF:8C:7C:C6:4C) ▂▄▆█
[D][wifi:434]: - 'BELL641' (48:29:52:FA:48:7E) ▂▄▆█
[D][wifi:434]: - 'yeelink-light-color2_miapCAC7' (04:CF:8C:7C:CA:C7) ▂▄▆█
[D][wifi:434]: - 'oxio-7410' (E8:2C:6D:50:74:14) ▂▄▆█
[D][wifi:434]: - 'WiFiGilles' (10:33:BF:E6:25:7E) ▂▄▆█
[D][wifi:434]: - '' (10:33:BF:E6:25:81) ▂▄▆█
[D][wifi:434]: - 'oxio-7410' (3C:90:66:F8:E8:64) ▂▄▆█
[D][wifi:434]: - 'HP-Print-36-ENVY 4500 series' (64:51:06:EC:E1:36) ▂▄▆█
[D][wifi:434]: - 'BELL872' (C0:3C:04:1B:FA:86) ▂▄▆█
[I][wifi:245]: WiFi Connecting to 'AJ - IOT'...
[V][wifi:247]: Connection Params:
[V][wifi:248]: SSID: 'AJ - IOT'
[V][wifi:251]: BSSID: FA:92:BF:5D:EF:A6
[V][wifi:271]: Password: NOPE
[V][wifi:276]: Channel: 5
[V][wifi:283]: Manual IP: Static IP=172.16.10.4 Gateway=172.16.10.1 Subnet=255.255.255.0 DNS1=0.0.0.0 DNS2=0.0.0.0
[V][wifi:287]: Hidden: NO
[V][wifi_esp32:434]: tcpip_adapter_dhcpc_stop failed: ESP_ERR_ESP_NETIF_DHCP_ALREADY_STOPPED
[E][wifi:291]: wifi_sta_connect_ failed!
[D][wifi:370]: Starting scan...
[V][wifi_esp32:647]: Event: WiFi Scan Done status=0 number=18 scan_id=178
[D][wifi:385]: Found networks:
[I][wifi:429]: - 'AJ - IOT' (FA:92:BF:5D:F4:37) ▂▄▆█
[D][wifi:430]: Channel: 5
[D][wifi:431]: RSSI: -53 dB
[I][wifi:429]: - 'AJ - IOT' (FA:92:BF:5D:EF:A6) ▂▄▆█
[D][wifi:430]: Channel: 5
[D][wifi:431]: RSSI: -38 dB
[D][wifi:434]: - 'AJ' (F4:92:BF:5D:EF:A6) ▂▄▆█
[D][wifi:434]: - 'AJ - 2.4Ghz' (FE:92:BF:5D:EF:A6) ▂▄▆█
[D][wifi:434]: - '' (02:92:BF:5D:EF:A6) ▂▄▆█
[D][wifi:434]: - ''Flashing the same config with Arduino seems to work fine, but wouldn't work OK with our i2c requirements.
krconv
commented
2 years ago I did hit an issue with connecting to the board remotely once (I've been uploading OTA), but I'm not sure if it was WiFi or not. Whatever it was, cutting power and restarting fixed it for me. Here is the full minimal config/console that I tried:
esphome:
name: home-power-monitor
esp32:
board: esp-wrover-kit
framework:
type: esp-idf
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
use_address: "192.168.86.223"
ota:
logger:
level: DEBUG
api:
[19:52:00][I][app:099]: ESPHome version 2021.11.4 compiled on Dec 1 2021, 19:50:35 [19:52:00][C][wifi:488]: WiFi: [19:52:00][C][wifi:350]: Local MAC: 40:F5:20:6A:26:68 [19:52:00][C][wifi:351]: SSID: [redacted] [19:52:00][C][wifi:352]: IP Address: 192.168.86.223 [19:52:00][C][wifi:354]: BSSID: [redacted] [19:52:00][C][wifi:355]: Hostname: 'home-power-monitor' [19:52:00][C][wifi:357]: Signal strength: -30 dB ▂▄▆█ [19:52:00][C][wifi:361]: Channel: 11 [19:52:00][C][wifi:362]: Subnet: 255.255.255.0 [19:52:00][C][wifi:363]: Gateway: 192.168.86.1 [19:52:00][C][wifi:364]: DNS1: 192.168.86.1 [19:52:00][C][wifi:365]: DNS2: 0.0.0.0 [19:52:00][C][logger:233]: Logger: [19:52:00][C][logger:234]: Level: DEBUG [19:52:00][C][logger:235]: Log Baud Rate: 115200 [19:52:00][C][logger:236]: Hardware UART: UART0 [19:52:00][C][ota:082]: Over-The-Air Updates: [19:52:00][C][ota:083]: Address: 192.168.86.223:3232 [19:52:00][C][api:134]: API Server: [19:52:00][C][api:135]: Address: 192.168.86.223:6053 [19:52:00][C][api:139]: Using noise encryption: NO [19:52:00][C][mdns:084]: mDNS: [19:52:00][C][mdns:085]: Hostname: home-power-monitor
Good call on deleting the build dir; I've noticed that some of the flags that change for the esp-idf library don't properly recompile all of the libraries.
I tested out that branch on my Vue, and the repeating task seemed to work first try. My notes
sensor.py to_code is missing the set_calibration to put the config value into C++I updated my branch here too; I copied some of your code (I'll figure out how to get the commits to show credit correctly later, committed them all as me to avoid the rebase/cherry-pick for now). Two notable difference
ct_clamp component as an example)I'm still waiting for the latest power readings from my power company; if they seem accurate, how would you guys feel about me opening a PR to the main repo? It seems pretty common for PRs to be open for a while with ESPHome and to make changes/give feedback, and I think it'll be good for visibility and to make it easier to for us and others review code with inline comments
Maelstrom96
commented
2 years ago So regarding my issue with the wifi component, it was a problem when using the manual_ip setting. I've submitted a PR to fix it and it has been merged.
The main mentality with esphome components seems to be to leave the data as raw as possible and change the behavior of the sensors using filters. For you data rate issue, you should be able to just setup a throttle filter. Also, a reason that we would want all those values is so that our total_daily_energy sensors are as accurate as possible. Even if you apply filters, the energy component will get the ~500ms state update and it will be able to calculate the most precise power consumption possible.
For the task I made, it should not be 100% blocking since other tasks should run when we're in vTaskDelay. Still need to investigate why ota wasn't working for you. I will be testing it locally.
Maelstrom96
commented
2 years ago A big problem I see with the way you do the loop is that the blocking/execution time will be way to high. There's also potential for even higher block time with this->read in cases where the read times out for whatever reason. I haven't found a way to change the default timeout of 1s yet.
krconv
commented
2 years ago Cool!
For the polling component vs publish on every reading; I don't think we are loosing any accuracy in our data by not publishing every value, they are still all accounted for in the average. People would need to add a filter to each of their CT sensors to reduce the frequency if they want the power data in Home Assistant (i.e. not just energy usage but current power load as well), or otherwise risk publishing too many updates. I think setting a single update_interval on the component is more intuitive, even if it isn't what is going on under the hoods; anyone could still set update interval to ~510ms if they want to speed up the readings.
I notice the priority on the RTOS task is set to 0; maybe reducing it would help (just a guess)? FWIW, the read loop is taking 23ms to read from I2c, which I think is ok given that this is working with specialized hardware and not intended for general purpose use. Aren't we taking the same amount of time with RTOS as well, or do they run on different cores? I'd argue giving ESP control over the loop would provide easier debugging, even if we occasionally get warnings that the loop took too long.
Got the most recent readings from my power company, and the values look pretty close to accurate as is! Mine were about 2% below the power company's readings, which I hope is only due to me not calibrating the phases exactly (using the default of 0.022 because I didn't get the calibration data before flashing). The missing sections are times of the day I was testing the Vue and readings weren't accurate
Maelstrom96
commented
2 years ago Regarding the task, the whole point was to offload the i2c read to the second CPU core, so it is running on the other core (cpu1). Also, I've just checked to validate the information, and the lower the priority number, the lower the priority is, so 0 should be the lowest one, which is the same priority as the idle task (see https://www.freertos.org/RTOS-task-priority.html)
Repurposing this issue for the discussion that it started. Was originally a question about connecting, and turned into a discussion about deciding the sensor data.
In the blog post, you mentioned that one of your USB-Serial adaptors wasn't working when you tried to connect to the ESP32. I'm wondering if I'm hitting the same issue; I only have one serial adaptor to test with at the moment though.
I'm using the FT232RL FTDI Mini USB to TTL Serial Converter, and getting nothing while trying to connect:
Process I'm taking to enter flashing mode:
But that's not working; my only thought is that maybe there is something else connected with TX/RX that is interfering? And a different serial converter could sort through the noise?
Any guidance would be appreciated!
Edit by @flaviut for visibility:
There is now a ESPHome component. You can follow development at