New firmware for EmonTX breaks the protocol

[hkapanen]

EmonTxV3, the OpenEnergyMonitor transmitter, is now shipping with a new firmware that changes the protocol:

Note: This firmware uses data whitening to improve the reliability of the data transmission. A standard pattern of 1s and 0s is overlayed on the underlying data, this prevents sync issues that result from too many zero values in a packet. When the packet is received it is decoded by emonhub. The 'whitening = 1' is required to tell emonhub to decode the packet correctly.

   #ifdef RF_WHITENING
       byte WHITENING = 0x55;
       for (byte i = 0, *p = (byte *)&tmp; i < sizeof tmp; i++, p++)
           *p ^= (byte)WHITENING;
   #endif

Some samples of the new firmware transmissions can be found in this Google groups thread.

Can you please add support for this as well.

[klohner]

Here's a first stab at an analysis. Was able to pull the data from these samples using this flex spec.

$ rtl_433 -R 0 -s 1024k -X n=emonTx,m=FSK_PCM,s=20,l=20,r=1000 -r g002_433.92M_1024k.cu8

However, g001 and g006 don't seem to contain the signal, and the messages returned by rtl_433 for the other cu8 files seem to contain a few incorrect bits in spots. I was able to filter the signal and pull clean messages with urh.

Here's a bitbench with a decoding that I modeled after RF12 packet format and design and RF12 broadcasts and ACKs documents.

Computing and comparing the CRC-16 on some of the older messages and these with reveng, it looks like the same polynomial.

$ reveng -w 16 -s 2dd2081a0700000000000000715cb80bb80bb80bb80bb80bb80b01000000399f 2dd2081a8404010035043d00ce5db80bb80bb80bb80bb80bb80b3d04000012a6 2dd2081a9304000029043a00615cb80bb80bb80bb80bb80bb80ba2b0000073cb
width=16  poly=0x8005  init=0x9e7f  refin=true  refout=true  xorout=0x0000  check=0xe1bc  residue=0x0000  name=(none)

$ reveng -w 16 -p 8005 -i 9e7f -l -c 2dd2081a0700000000000000715cb80bb80bb80bb80bb80bb80b01000000
399f

$ reveng -w 16 -s 2dd20f285a555555fb34555555555555555555555555555555555555555555555555652065206520555555557f22 2dd20f2845555555e83455555555555555555555555555555555555555555555555565206520652055555555d8d6 2dd20f2844555555dc34555555555555555555555555555555555555555555555555652065206520555555557243 2dd20f28475555553e34555555555555555555555555555555555555555555555555652065206520555555551e5b 2dd20f2846555555bc34555555555555555555555555555555555555555555555555652065206520555555552406 
width=16  poly=0x8005  init=0x9e7f  refin=true  refout=true  xorout=0x0000  check=0xe1bc  residue=0x0000  name=(none)

$ reveng -w 16 -p 8005 -i 9e7f -l -c 2dd20f285a555555fb3455555555555555555555555555555555555555555555555565206520652055555555
7f22

The old style messages decode with rtl_433 -y if we append the normal expected 0x555555 prefix and '0xaa' suffix.

$ rtl_433 -R 45 -y {288}5555552dd2081a0700000000000000715cb80bb80bb80bb80bb80bb80b01000000399faa
model     : emonTx-Energy node      : 08           ct1       : 7             ct2       : 0             ct3       : 0             ct4       : 0             batt_Vrms : 236.65
pulse     : 1

However, these new whitened messages come with a slightly different prefix of 0x555555aa with the expected 0xaa suffix. It may be that the 0x555555aa is some kind of flag to say that the message is whitened. The whitening seems to be repeating 0x55 XOR'd just to the payload.

I tried to trick rtl_433 to decode the whitened g002 sample by de-whitening the payload, recomputing the CRC-16, and adding the old-style 0x555555prefix and 0xaasuffix, but this was not successful. The current driver doesn't provide - vv messages, but it seems like it's picky about the payload being exactly 26 (0x1a) bytes to decode fields, and not the 40 (0x28) byte payload specified in the header and found in these new messages.

$ reveng -w 16 -p 8005 -i 9e7f -l -c 2dd20f285a555555fb3455555555555555555555555555555555555555555555555565206520652055555555
7f22

WHITENED PAYLOAD: 5a555555fb3455555555555555555555555555555555555555555555555565206520652055555555
WHITENING:        55555555555555555555555555555555555555555555555555555555555555555555555555555555
XOR'd PAYLOAD:    0f000000ae6100000000000000000000000000000000000000000000000030753075307500000000

$ reveng -w 16 -p 8005 -i 9e7f -l -c 2dd20f280f000000ae6100000000000000000000000000000000000000000000000030753075307500000000
5d9c

rtl_433 -vv -R 45 -y {400}5555552dd20f280f000000ae61000000000000000000000000000000000000000000000000307530753075000000005d9caa
[no result]

There seems to be a good bit of work needed in order to get the driver to recognize and decode these new whitened signals. I don't have the bandwidth right now to do this.

[klohner]

Figured out a bit more.

First, the CRC-16 does not include the 0x2d sync byte in its computation. Removing this before feeding to reveng changes the CRC-16 initial value to 0xffff, which with polynomial 0x8005 (or reversed as 0xa001 for how _crc16_update() knows it) gives us a recognized CRC format of "crc-16/modbus".

$ reveng -w 16 -s d20f285a555555fb34555555555555555555555555555555555555555555555555652065206520555555557f22 d20f2845555555e83455555555555555555555555555555555555555555555555565206520652055555555d8d6 d20f2844555555dc34555555555555555555555555555555555555555555555555652065206520555555557243 d20f28475555553e34555555555555555555555555555555555555555555555555652065206520555555551e5b d20f2846555555bc34555555555555555555555555555555555555555555555555652065206520555555552406
width=16  poly=0x8005  init=0xffff  refin=true  refout=true  xorout=0x0000  check=0x4b37  residue=0x0000  name="CRC-16/MODBUS"

$ reveng -m crc-16/modbus -c d20f285a555555fb3455555555555555555555555555555555555555555555555565206520652055555555
7f22

I took a look at the older EmonTx3 firmware and the newer EmonTxV3CM firmware and see that besides the whitening, the message structure has changed. The first byte of the HEAD section is the Node ID. Depending on this ID, the format of the PAYLOAD could be one of several different formats.

There's info on most all of the older types, including the "EmonTx v3" 0x08 format which rtl_433 currently supports, in the EmonHub Configuration document. This newest "EmonTxV3" format (which includes the data whitening feature) is shown in comments near the top of its firmware source code.

Using all this info, I was able to take the de-whitened payload from the g002 sample (which has a NodeID of 15 (0x0f), reformat it like the older EmonTx v3 NodeID 0x08 format, compute a CRC, and feed it to rtl_433 for parsing.

# Orignal g002 message:
# {408}555555aa2dd20f285a555555fb34555555555555555555555555555555555555555555555555652065206520555555557f22aa
# KEY: ~~~~~~~~SSGGNNLLPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPPCCCC~~
#   ~ = prefix and suffix of signal, typically 0x55 or 0xaa to help get bitrate
#   S = SYNC (0x2d is what the HOPE RF12B and RF69CW chips like)
#   G = Group (0xd2 (int 210) seems to be hardcoded into the EmonTx3 source)
#   N = Node ID (helps identify the EmonTX payload format)
#   L = Length of Payload in bytes (up to 66 bytes as defined by jeelabs R12)
#   P = Payload (format of this is typically determined by the NodeID, may be "whitened" by XOR'd with 0x55)
#   C = CRC-16 (poly=0x8005  init=0xffff) "CRC-16/MODBUS" for Group through Payload bytes

Node ID:           0f

Payload Format for Node ID 15:
  nodename = EmonTxV3CM_15
  [[[rx]]]
    names = MSG, Vrms, P1, P2, P3, P4, E1, E2, E3, E4, T1, T2, T3, pulse
    datacodes = L,h,h,h,h,h,L,L,L,L,h,h,h,L
    scales = 1,0.01,1,1,1,1,1,1,1,1,0.01,0.01,0.01,1
    units = n,V,W,W,W,W,Wh,Wh,Wh,Wh,C,C,C,p

PAYLOAD:               5a555555fb3455555555555555555555555555555555555555555555555565206520652055555555
XOR'd PAYLOAD:         0f000000ae6100000000000000000000000000000000000000000000000030753075307500000000
KEY:                   MSG-----VrmsP1--P2--P3--P4--E1------E2------E3------E4------T1--T2--T3--PULSE---

Payload Format for Node ID 8:
  nodename = emonTx_3
  firmware =V2_3_emonTxV3_4_DiscreteSampling
  hardware = emonTx_(NodeID_DIP_Switch1:OFF)
  [[[rx]]]
     names = power1, power2, power3, power4, Vrms, temp1, temp2, temp3, temp4, temp5, temp6, pulse
     datacodes = h,h,h,h,h,h,h,h,h,h,h,L
     scales = 1,1,1,1,0.01,0.1,0.1, 0.1,0.1,0.1,0.1,1
     units =W,W,W,W,V,C,C,C,C,C,C,p

KEY:                   P1--P2--P3--P4--VrmsT1--T2--T3--T4--T5--T6--PULSE---
Reformatted PAYLOAD:   0000000000000000ae6130753075307530753075307500000000
with GRP + HEAD: d2081a0000000000000000ae6130753075307530753075307500000000

$ reveng -m crc-16/modbus -c d2081a0000000000000000ae6130753075307530753075307500000000
9775

$ rtl_433 -R 45 -y {288}5555552dd2081a0000000000000000ae61307530753075307530753075000000009775aa
model     : emonTx-Energy node      : 08           ct1       : 0             ct2       : 0             ct3       : 0             ct4       : 0
batt_Vrms : 250.06       pulse     : 0             temp1_C   : 3000.0        temp2_C   : 3000.0        temp3_C   : 3000.0        temp4_C   : 3000.0
temp5_C   : 3000.0       temp6_C   : 3000.0        Integrity : CRC

The spurious temperature readings highlight something interesting. The firmware source code implements some special temperature values to denote a few special cases. In emonLibCM.h we find:

#define UNUSED_TEMPERATURE 30000     // this value (300C) is sent if no sensor has ever been detected
#define OUTOFRANGE_TEMPERATURE 30200 // this value (302C) is sent if the sensor reports < -55C or > +125C
#define BAD_TEMPERATURE 30400        // this value (304C) is sent if no sensor is present or the checksum is bad (corrupted data)
                                     // NOTE: The sensor might report 85C if the temperature is retrieved but the sensor has not been commanded
                                     //  to measure the temperature.

The driver does know about the 300C special case (which was used in the older 0x08 Node ID), but it's not picked up because the new 0x0f Node ID protocol defaults temperature values to be 0,01 value, instead of 0,1 value as seen in the 0x08 Node ID protocol. So, for sake of completeness to force the new data into the old format:

300C * 100 = 30,000 (0x7530)
300C *  10 =  3,000 (0x0bb8)

So change: d2081a0000000000000000ae6130753075307530753075307500000000
To:        d2081a0000000000000000ae61b80bb80bb80bb80bb80bb80b00000000

$ reveng -m crc-16/modbus -c d2081a0000000000000000ae61b80bb80bb80bb80bb80bb80b00000000
5d6e

$ rtl_433 -R 45 -y {288}5555552dd2081a0000000000000000ae61b80bb80bb80bb80bb80bb80b000000005d6eaa
model     : emonTx-Energy node      : 08           ct1       : 0             ct2       : 0             ct3       : 0             ct4       : 0
batt_Vrms : 250.06       pulse     : 0

And finally, there doesn't seem to be any flag to say that the payload has been whitened. Although whitening is the default and should probably be assumed if Node ID is 15 or 16. However, perhaps an extremely high power reading (like above 16,383 watts) would be a good indicator that the entire payload should have been de-whitened first.

[hkapanen]

Bump -- could anyone help on this?

[gdt]

What's the status? it seems this is understood, so the question is how to modify a decoder, or create a new one, to recognize the new version, while also still recognizing the new one. Is this still needed with the current code - a lot has been committed in 3 years.

[Nauttor]

I upgraded an old system where I was running the old version of RTL_433 and now my old emontxv3 is not working anymore with the rtl_433. It seems whatever was change in the rtl_433 is not compatible anymore to read what the emontxV3 with the old firmware is sending. I tried to upgrade the Emontxv3 to the latest firmware available from 2023 and still no reads in the 22.11 version.

I tried with the mentioned lines before but the information recovered back is imcomplete.

$ rtl_433 -R 45 -y {288}5555552dd2081a0000000000000000ae61b80bb80bb80bb80bb80bb80b000000005d6eaa rtl_433 version 22.11 (2022-11-19) inputs file rtl_tcp RTL-SDR SoapySDR

---

time : 2024-06-25 17:58:21 model : emonTx-Energy node : 08 ct1 : 0 ct2 : 0 ct3 : 0 ct4 : 0 batt_Vrms : 250.06 pulse : 0 Integrity : CRC

My emontx is sending node id, which is identfied, ct1,2,3,4, and also temperature, the Ct1,2 both appeared empty and the temp is not even recognized.

Did someone get to make it work?

[gdt]

@Nauttor your report is very vague. You said "old version" and you didn't specify it. 22.11 is an old version. Please say what versions really. Please also build yourself from git master and test that. Also specify OS, OS version, CPU type, rltsdr version, dongle hardware,

[Nauttor]

@Nauttor your report is very vague. You said "old version" and you didn't specify it. 22.11 is an old version. Please say what versions really. Please also build yourself from git master and test that. Also specify OS, OS version, CPU type, rltsdr version, dongle hardware,

Hi @gdt,

Sorry for my vague report, reading it you are absolutely right.

So going back to the issue.

My old system was an Atom x86 running an old debian, proabably a 9, and the version I built on the old system was rtl_433 version 20.02-99-g50c774d branch master at 202007202001 at the time.

That version was workinf fine with the Emontx decoder until last month when my old system died.

I started fresh on a new system xeon x86 and I tried with the included package in debian 12, being rtl_433 22.11 and I trid today as you suggested build it form source.

About the firmware running in my emontx I was running a very old version which I upgraded yesterday to the latest available for my hardware, emontx V3 , but that was a pointless effort as the result working against the 22.11 or the latest 23.11-140-g367378fa the result is the same no data decoded.

This is an extract of the latest version where you can see also my Radio hardware reflected.

rtl_433 version 23.11-140-g367378fa branch master at 202406230725 inputs file rtl_tcp RTL-SDR with TLS Detached kernel driver Found Rafael Micro R820T tuner [SDR] Using device 0: Realtek, RTL2838UHIDIR, SN: 00000001, "Generic RTL2832U OEM" Exact sample rate is: 250000.000414 Hz [R82XX] PLL not locked!

I can confirm also the radio is working as I continue to receive 2 other temperature sensors when I use it with decoders -R 42 and 8.

If I tried the suggested line before I received an incomplete view of the data received. Incomplete as I don't recieve temperature data and Ct1 and 2 also should contain data. Vrms and node Id seems fine.

~/rtl_433$ rtl_433 -R 45 -y {288}5555552dd2081a0000000000000000ae61b80bb80bb80bb80bb80bb80b000000005d6eaa rtl_433 version 23.11-140-g367378fa branch master at 202406230725 inputs file rtl_tcp RTL-SDR with TLS

---

time : 2024-06-26 10:13:34 model : emonTx-Energy node : 08 ct1 : 0 ct2 : 0 ct3 : 0 ct4 : 0 batt_Vrms : 250.06 pulse : 0 Integrity : CRC debian@Multi:~/rtl_433$

I hope this bring some more light, let me know if I need to provide more data.

[zuckschwerdt]

The change history is listed here https://github.com/merbanan/rtl_433/commits/master/src/devices/emontx.c I don't see any effective changes since your previous 2020-07-20 version.

The rtl_433 line you used above does not receive but only decode the given code. That test code won't contain any current data.

Do you still receive codes but decoding then fails? Try rtl_433 -R 45:v

Otherwise can you try to capture a sample from that sender? https://triq.org/rtl_433/ANALYZE.html

[Nauttor]

Hi,

I tried this rtl_433 -R 45:v but nothing pops up, like is nothing is received. EDIT: I did it again and seems the :v is not accepted by the protocol 45. Protocol [45] "emonTx OpenEnergyMonitor" does not take arguments "v"!

I did the capture of what seems to be the traffic of the device as it is send each 10 seconds.

Also it seems they have always the same size.

Jun 26 14:20 g007_433.92M_250k.cu8 Jun 26 14:20 g008_433.92M_250k.cu8 Jun 26 14:21 g009_433.92M_250k.cu8

This is one of the generated files.

Test mode active. Reading samples from file: g007_433.92M_250k.cu8 ;ook 2 pulses ;freq1 -9880 ;samplerate 250000 Hz ;rssi -0.1 dB ;snr 12.0 dB ;noise -12.1 dB 284 44 1500 15004 ;end

Test mode active. Reading samples from file: g007_433.92M_250k.cu8 Detected OOK package @0.179776s Analyzing pulses... Total count: 2, width: 1.83 ms ( 457 S) Pulse width distribution: [ 0] count: 1, width: 284 us [284;284] ( 71 S) [ 1] count: 1, width: 1500 us [1500;1500] ( 375 S) Gap width distribution: [ 0] count: 1, width: 44 us [44;44] ( 11 S) Pulse period distribution: [ 0] count: 1, width: 328 us [328;328] ( 82 S) Pulse timing distribution: [ 0] count: 1, width: 284 us [284;284] ( 71 S) [ 1] count: 1, width: 1500 us [1500;1500] ( 375 S) [ 2] count: 1, width: 44 us [44;44] ( 11 S) [ 3] count: 1, width: 15004 us [15004;15004] (3751 S) Level estimates [high, low]: 16021, 1012 RSSI: -0.1 dB SNR: 12.0 dB Noise: -12.1 dB Frequency offsets [F1, F2]: -2590, 0 (-9.9 kHz, +0.0 kHz) Guessing modulation: Pulse Width Modulation with fixed gap view at https://triq.org/pdv/#AAB104011C05DC002C3A9C829355 Attempting demodulation... short_width: 284, long_width: 1500, reset_limit: 48, sync_width: 0 Use a flex decoder with -X 'n=name,m=OOK_PWM,s=284,l=1500,r=48,g=0,t=486,y=0' pulse_demod_pwm(): Analyzer Device bitbuffer:: Number of rows: 1 [00] { 2} 80 : 10

Applying the suggested flex decoder I see this

rtl_433-20.11$ rtl_433 -X 'n=name,m=OOK_PWM,s=284,l=1500,r=48,g=0,t=486,y=0' g007_433.92M_250k.cu8 rtl_433 version unknown inputs file rtl_tcp RTL-SDR Use -h for usage help and see https://triq.org/ for documentation. Trying conf file at "rtl_433.conf"... Trying conf file at "/home/debian/.config/rtl_433/rtl_433.conf"... Trying conf file at "/usr/local/etc/rtl_433/rtl_433.conf"... Trying conf file at "/etc/rtl_433/rtl_433.conf"... Registered 146 out of 175 device decoding protocols [ 1-4 8 11-12 15-17 19-21 23 25-26 29-36 38-60 63 67-71 73-100 102-105 108-116 119 121 124-128 130-149 151-161 163-168 170-175 ] Test mode active. Reading samples from file: g007_433.92M_250k.cu8

---

time : @0.179776s model : name count : 1 num_rows : 1 rows : len : 2 data : 8 codes : {2}8

[zuckschwerdt]

does not take arguments "v"!

Older version don't support per-decoder verbosity. Maybe use rtl_433 -vv -R 45

Also it seems they have always the same size.

That is expected, recordings are always full receive frames.

Verfiy a good file with https://triq.org/pdv/ then upload one as zip here. The OOK demod is not applicable, we want FSK.

[Nauttor]

g007_433.92M_250k.cu8.zip

I checked it in triq.org, but not sure what to look for there honestly.

rtl_433 -vv -R 45 this does not return nothing also.

[klohner]

That sample looks to be rather weak and was recorded at too low of a bit rate. It may also be the case that the second frequency in the FSK was outside the bandwidth and wasn't captured. It's difficult to pull out clean data from the background.

@Nauttor - I'd suggest you try to capture this signal closer to its source, and be sure to use -s 1000k or -s 2000k in the command line to specify a higher sample rate.

[Nauttor]

HI @klohner,

I will try tomorrow and I will disconnect the other 433 devices I have to avoid any noise from the others.

[Nauttor]

Just to provide a proper closure. It took me some time to test it properly but it seem to be working fine now. The problem was not the version, as with the latest firmwares in the latests version and the latest github rtl433 all works as expected. ONly thing that makes the difference was these -s 1000k or -s 2000k. Without opening the sample rate it doe snot work and the data is missed. Not sure if that is specific to my radio or if it need to be considered to be modified in the code.

Thanks for your help