Last byte is a crc checksum

[baryluk]

Was a little bored, and was sure last byte (data[63]) is some kind of checksum, because its distribution is so random. I didn't verify the same data[63] is for same data[0:-1], but I was sure it is a deterministic checksum. Using crc package, I tried few pre-existing crc methods, and they didn't work. I also tried simple xor and 8 and 16-bit modulo adds, with and without first byte. Didn't work.

Then used wonderful reveng by Gregory Cook, dumped a bit of data to files, and found the correct crc in milliseconds:

user@debian:~/Downloads/reveng-3.0.5$ time ./reveng -w 8 -s $(shuf -n 100 /tmp/megadump2.txt)
width=8  poly=0x39  init=0x42  refin=false  refout=false  xorout=0x00  check=0x4b  residue=0x00  name=(none)

real    0m0.011s
user    0m0.007s
sys 0m0.004s
$

import crc  # pip install crc

def main():
   ...

    # $ ./reveng -w 8 -s $(shuf -n 100 /tmp/dump2.txt)
    # width=8  poly=0x39  init=0x42  refin=false  refout=false  xorout=0x00  check=0x4b  residue=0x00  name=(none)
    def setup_crc():
        width = 8
        poly = 0x39
        init_value = 0x42
        final_xor_value = 0x00
        reverse_input = False
        reverse_output = False
        configuration = crc.Configuration(width, poly, init_value, final_xor_value, reverse_input, reverse_output)
        use_table = True
        crc_calculator = crc.CrcCalculator(configuration, use_table)
        return crc_calculator

    crc_calculator = setup_crc()

....

    def decode(data):
        ....

        # print(''.join(['{:02x}'.format(x) for x in data]))  # for feeding to reveng
        # return

        actualChecksum = data[-1]
        expectedChecksum = crc_calculator.calculate_checksum(data[0:-1])
        if actualChecksum != expectedChecksum:
           print(f"Ignoring packet of length {len(data)} with unexpected checksum. Expected: {expectedChecksum:02x} Actual: {actualChecusum:02x}", file=sys.stderr)
           return

Will do some more testing, and commit changes later.

[baryluk]

@didim99 Just so you know. I figured out what is the last byte ;)

You figured what is the first byte. So we are now well covered!

[didim99]

@baryluk nice work! Checksum in the tail of packet is a standard practice for various binary protocols and in most cases it is so. I also tried some standard algorithms to calculate checksum for this packets but failed. Thank open source world for very useful tools what makes our life easer, I'll take on board this.

[baryluk]

AFAIK These power meters use STM32Fx, and they do have hardware support for 8, 16 and 32-bit CRCs, with fully programmable options (custom polynomials, init, reversals), and found some pdf docs and examples on the internet. Unfortunately because they are fully programmable, devs could have used any polynomal they want. I checked polynomial 0x39 and it is not listed in any of crc polynomial zoos or selection of good polynomials. I run Philip Koopman's hdlen.cpp on it, and it does show it is a bad polynomial compared to other ones:

$ ./hdlen 0x39 1 7
Recompile with -DOPTZ for fast version
Poly=0x39 startHD=3 maxHD=6
# 0x39  HD=3  len=57  Example: Len=58 {0} (0x20) (Bits=2)
# 0x39  HD=4  len=5  Example: Len=6 {0,3} (0x20) (Bits=3)
# 0x39  HD=5  len=1  Example: Len=2 {0} (0x25) (Bits=4)
# 0x39  HD=6  NONE  Example: Len=1 {0} (0x39) (Bits=5)
0x39 {57,5,1}

It has a Hamming distance of only 5 with len 1. And Hamming distance of 3 with len 57. data words we have here are len 63, so in fact this crc cannot correct any bit in the input deterministically. It can still do kind of a verification of course, which we want here. Optimal polynomial would be 0xa7 or 0x9b for this use case, but I guess optimality and correction function of crc was not in the mind of devs when they selected random polynomial.

[baryluk]

Just for reference. I use crc package, but another Python library, crccheck (this one is very versatile too and is packaged in debian so might switch to it later), mentions 0x39: https://github.com/MartinScharrer/crccheck/blob/main/crccheck/crc.py#L739-L749

It mentions 0x39 as CRC-8/DARC

[baryluk]

crccheck got these CRCs auto-generated from reveng website actually.

It is indeed listed there https://reveng.sourceforge.io/crc-catalogue/1-15.htm :

CRC-8/DARC

width=8 poly=0x39 init=0x00 refin=true refout=true xorout=0x00 check=0x15 residue=0x00 name="CRC-8/DARC"

    Class: attested
    The single codeword is supported by the codewords confirming [CRC-6/DARC](https://reveng.sourceforge.io/crc-catalogue/1-15.htm#crc.cat.crc-6-darc), defined identically apart from Poly in the same standard.
    ETSI [EN 300 751](https://www.etsi.org/deliver/etsi_en/300700_300799/300751/01.02.01_60/en_300751v010201p.pdf) version 1.2.1 (January 2003)
        I Definition: Width, Poly (Section 11.2.3, p.68)
        I Definition: RefIn, RefOut (Section 12, pp.69–70)
        IV 1 codeword (Section 11.2.3, p.68)
            00000001​00000011​11010111
        See section 12 for details of the transmission order.
    Created: 14 December 2009
    Updated: 29 November 2018

[baryluk]

Init value is different, so maybe this is just a coincidence.