Feature request: Extract/preserve power meter calibration.

[dwmw2]

I have a bunch of Tongou DIN rail power meters (https://www.elektroda.com/rtvforum/viewtopic.php?p=21191153#21191153),. and have finally got round to liberating them.

I've been experimenting with them, and found that the calibration parameters for the BL0942 chip are available both in the DPs that the Tuya firmware reports, and also in the flash. More details at https://github.com/esphome/issues/issues/6155

So apart from the one I already converted without reading its flash first, I can liberate them without losing the factory calibration. It would be nice if the tooling could help new users get this right.

Example firmware dump at http://david.woodhou.se/tongou-tuyafw.bin (sha256 a0b0a2804946f8760f5be0360a3b0325d204087c6aae6c7917297cba2f1341ba)

I found the calibration at 0x001d5000:

001d5000  60 3e 00 00 82 30 00 00  13 0c 00 00 5d 0a 00 00  |`>...0......]...|

That's

• UREF = 0x3e60 = 15968
• IREF = 0x3082 = 12418 (times ten to make 124180)
• PREF = 0xc13 = 3091 (divide by ten to make 309.1)
• EREF = 0xa5d = 2653

We can apply a sanity check to those, as the PREF can be derived from UREF/IREF, and EREF from PREF as described in the ESPHome ticket above.

[kuba2k2]

Wow, this is huge.. I had no idea that the calibration is stored in the flash!

Well, in fact, I didn't look at the BL0942/BL0937 drivers at all. All I know is that there are always calibration problems, especially on BL0937 when using it in ESPHome. Perhaps this one has calibration stored in flash somewhere too?

Assuming that the BL component in ESPHome would allow for calibration in YAML, ltchiptool should be able to pull the config data from flash - when running Kickstart, it has full access to the flash readouts. That could finally make the default BL config usable...

The one you converted without reading its flash - is the calibration still stored there? Or did it get overwritten by ESPHome?

[dwmw2]

I have a patch to make the BL0942 driver in ESPHome take the calibration in YAML: https://github.com/dwmw2/esphome/commit/ccc60a54d9dd76070d827651cc4d084f51d29054

I already have an outstanding PR to fix the init sequence, so I'm waiting for that to be merged before submitting. (Was also still working things out and testing, but I think I'm happy with it now).

[dwmw2]

And no, the unit for which I lost the calibration was the one in https://github.com/libretiny-eu/ltchiptool/issues/37 and the relevant part of the flash hadnalready been overwritten by the time I took the dump there.

But I do have valid calibration data for a bunch of other units, which are all fairly similar, so I can do iref/uref manually for this one and have confidence in the result being sane.

TBH the calibration may have changed when I opened it up and the coil sits differently now anyway. Seriously tempted to do keyhole surgery for the rest of the units, as they can be flashed with SOIC probes just on the GND/Vcc/TX/RX pins which all come to the BL0942.

[kuba2k2]

Ah, right, so installing Kickstart overwrites the calibration already? That would mean it's not possible to do automatic config generation here, not that easily at least.

[dwmw2]

Ah, right, so installing Kickstart overwrites the calibration already?

I wouldn't swear to that. I didn't just flash the Kickstart on this unit. It was already dead, which is why I took it out of service and opened it up. Then tried to flash ESPHome, then OpenBK before filing #37 in which you pointed out the bootloader was hosed.

I have plenty of clean units to test with.

This region of the flash is right after the JSON which describes which pins do what; surely Kickstart wants to preserve that too?

And anyway, these units aren't remotely exploitable so it's all being flashed by ltchiptool, and surely we can read what's there before we overwrite it?

[dwmw2]

My ESPHome config for these is at https://git.infradead.org/?p=users/dwmw2/esp32-pool.git;a=commitdiff;h=514c81a607be324c5952fe636b7b86397ad782de btw.

[kuba2k2]

I looked at the partition table, 0x1D5000 is in the "key value store" partition, which is used by ESPHome for storing configuration data - so yes, it will get overwritten by Kickstart or ESPHome.

This JSON you found is actually not used by Kickstart at all - it isn't present in all devices. Instead, the same JSON-like value is kept in encrypted Tuya storage, by the end of the flash. Hence Kickstart can still read it, because it's not used by ESPHome.

Yes, you can read flash with ltchiptool before flashing custom firmware, and you definitely should :slightly_smiling_face:

A full pre-Kickstart firmware dump could provide the calibration data to UPK2ESPHome.

[dwmw2]

I seem to have two units (out of twelve) which just report zero for all four DPS values. I'll take a proper firmware dump and experiment with them when I take them out of production to 'liberate' them.

[dwmw2]

Now documented: https://devices.esphome.io/devices/Tongou-TO-Q-SY1-JWT

[kuba2k2]

Note that the message shown in flashing section comes from BK7231 GUI tool, not ltchiptool. Otherwise nicely documented, with code snippets and what not.

[dwmw2]

Thanks. I'll fix that.

The "if it needs documenting, fix it" principle applies, of course. Which is what this ticket is about :)

[dwmw2]

Dump and serial output from one of those all-zeros units. Was hoping to see it print one of the messages about the coefficients that I see in the app dump, but no luck.

http://david.woodhou.se/tuya-firmware-output-a0920823d9b6.txt http://david.woodhou.se/tuya-firmware-a0920823d9b6.bin

[dwmw2]

I've now finished 'liberating' the rest of the devices in my consumer unit. The address isn't always 0x1d5000. It varies. For example in this one it's at 0x001d700:

001d5000  7b 22 6d 61 78 22 3a 30  2c 22 6c 61 73 74 22 3a  |{"max":0,"last":|
001d5010  30 2c 22 66 69 72 73 74  22 3a 30 7d 00 00 00 00  |0,"first":0}....|
001d5020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
001d5040  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
001d6000  7b 22 6d 61 78 22 3a 30  2c 22 6c 61 73 74 22 3a  |{"max":0,"last":|
001d6010  30 2c 22 66 69 72 73 74  22 3a 30 7d 00 00 00 00  |0,"first":0}....|
001d6020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
001d6040  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
001d7000  3c 3e 00 00 9c 31 00 00  47 0c 00 00 89 0a 00 00  |<>...1..G.......|
001d7010  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
001d8000  01 ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
001d8010  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*

Any thoughts on how we might find it automatically?

[ianchi]

Hi @dwmw2,

I've been doing some digging in my extracted firmwares. All four of them have the values at offset 0x1d7000.

All in the same range of coefficients as yours

[ianchi]

@dwmw2 where you able to test current readings against some reference? The calibration factor in flash for current is almost half than the default in ESphome. This will make current readings double, isn’t it?

That will make very strange figures for some of my appliances (too large).

[ianchi]

My bad.

It seems that now I’m effectively measuring half the real value.

I found a reference in a smart air conditioner (in blue) and it’s around double of my current ESPHome Tongou (purple)

Great findings @dwmw2 (very bad news for my consumption though)…

[dwmw2]

I deliberately left the ESPHome defaults alone when I added the calibration support, but they don't make much sense. IIRC the current readings are low but the power is closer to reality, because they aren't even consistent? Although the Tuya power factor is always a tiny bit lower than the calculations suggest too.

[szupi-ipuzs]

Just to add my two cents: I found calibration in my tongou din relay at 001d7000