TFA 30.3233.01 support

[Fietspomp86]

Hi guys,

I have a new rain-sensor, the TFA 30.3233.01. Website:

Could I please get some guidance how-to add this? I tried capturing the raw packages but I didn't succeed. Running a Pi3 with an RTL dongle. I see plenty of sensors from the neighbours to it's working. Just need some help with this new sensor.

[zuckschwerdt]

Capturing the signal is indeed the first step.

• Use -S all to capture a few packets, then read them back with -r mode and sort out the known ones (or maybe use -S unknown).
• Verify the samples visually with http://triq.net/iqs
• Use -A mode to get an idea what's in the samples.

Post your findings and upload a zip with some samples known to be from the device. A good way to only capture that device is to remove the antenna and place the device about 10 cm from the recevier.

[Fietspomp86]

I did as you suggested, placed the device very close to the pi, removed antenna of the dongle. I believe I caught something, but I'm no expert. Without the device near the dongle I tried to scan and found nothing, so I guess it's really only the rainmeter which was in range.

RTL-433.zip

[zuckschwerdt]

It is always good to add some length of USB cable between the Raspi and the dongle. Plugged directly into the ports it will get noticable noise.

[Fietspomp86]

Ok, I used an extension USB cable and tried again. Here's the sample zip: sample.rain.zip

[mhaas]

Hi @Fietspomp86, I am also looking into this device! I have many samples and some idea how to decode. I will keep you posted.

[mhaas]

Hi all,

I gathered some samples and loaded one into sigrok. It seems that the signal has a period of 750us. At the beginning of a period, a pulse is sent at either 250us or 500us length.

The sensor sends the same message 6 times. At the beginning of each message, two sync pulses are sent: 750us high, 750us low, 750us high, 750us low. Note that the screenshot shows the beginning of the very first message, where only one sync pulse (750us hgh, 750us low) is sent.

@zuckschwerdt Does this analysis make sense to you?

The modulation seems to be PWM. Given this data, the following flex decoder:

rtl_433 \
      -X 'name=tfa_drop,modulation=OOK_PWM,short=255,long=510,reset=2500,gap=1300,sync=750,bits=66'

produces output:

{
  "time": "2019-12-31 12:26:13",
  "model": "tfa_drop",
  "count": 6,
  "num_rows": 7,
  "rows": [
    {
      "len": 1,
      "data": "8"
    },
    {
      "len": 66,
      "data": "c65d54ff09550047c"
    },
    {
      "len": 66,
      "data": "c65d54ff085500dc4"
    },
    {
      "len": 66,
      "data": "c65d54ff085500dc4"
    },
    {
      "len": 66,
      "data": "c65d54ff085500dc4"
    },
    {
      "len": 66,
      "data": "c65d54ff085500dc4"
    },
    {
      "len": 66,
      "data": "c65d54ff085500dc4"
    }
  ],
  "codes": [
    "{1}8",
    "{66}c65d54ff09550047c",
    "{66}c65d54ff085500dc4",
    "{66}c65d54ff085500dc4",
    "{66}c65d54ff085500dc4",
    "{66}c65d54ff085500dc4",
    "{66}c65d54ff085500dc4"
  ]
}

Note that the first row of len 1 is an artifact of the first sync pulse.

I have already gathered many samples, including a nice rain simulation by putting the sensor under a running sink for an hour or so. I find it most intuitive to look at the data at the crumble level (two crumbles are one nibble).

Here is a histogram:

This shows the relative freqencies of each crumble value at a given index in the data. Note that I am adding a padding of a 0 nibble on the right.

What is immediately obvious here is that the first three bytes (crumbles 0 to 11) are always identical. This is the device ID - it changes after a battery reset. The histogram above was generated from just one run, so the ID is the same.

From looking at the data itself, I know that crumbles 16-19 form the LSB of the rain counter. The data is not entirely clean. I assume some sort of transformation has to be applied - but looking at the bits as they change, it is clear that lower bits roll over and higher bits increase.

Interestingly, after 8 tips of the rocker, crumbles 16, 17 and 24-27 flip from 00 to 11. So 24-27 also seem to be related to the rain counter, which would give us 8 crumbles or 2 bytes to encode the rain data.

I will investigate a bit more. It seems that I need to XOR crumbles 18 and 19 with 0111 to get sensible data, but I don't know what to do with the other bits.

[zuckschwerdt]

The analysis on crumbles and the histogram is great. I have an old half-finished tool to detect change frequency, roll-over and similar. But it was a rabbit hole. It seemed I always needed to add too many assumptions. Better to just show a picture and squint your eyes ;)

The unexpected double bit flip might be a parity bit. Some protocols have 7-to-8 encoding.

[mhaas]

@zuckschwerdt Well, I have a script which dumps only the changed crumbles. High-entropy columns are bold, low-entropy columns are light. Sadly, the output does not transport very well except as a screenshot.

The leftmost column here is the number of tips of the rocker which 0000 being the initial state and 0001 the first tip. Note that some values are duplicated.

It is easy to see that crumble 19 changes often, then crumble 18. It also seems obvious that 00 is the highest possible value, after which the next significant bit increases. 11 is then the value after rollover, so likely the smallest possible value. This seems to hold true for both crumble 18 and 19.

I am still confued by what's happening after 0009 - and why the initial state at 0000 does not seem to correspondly nicely to something zero-ish.

I was also looking at BCD, but that does not seem to be case. Perhaps some gray code or hamming code. I will check later.

I also have some more data for 50, 75 and 100:

[zuckschwerdt]

If I read that right and 00 is assumed to be the biggest and 11 the smallest value that might indicate the bits are flipped? Though I guess you already looked into that. There is one other decoder that does not roll over a counter to zero. IIRC it divides by a large factor and keeps the remainder.

[mhaas]

I think I got it. You were right about flipping the bits. The missing transformation is the addition of 10 (as in, ten). This is somewhat obvious in the bitbench (cool tool, btw!) after inverting: Bitbench

Basically, zero is at 65526 - 9 is at 65535, and thus 10 (ten) must be at 0.

The following python code implement this transformation:

def simple_decoder(input_data) -> int:
    rain_a = input_data.get_byte_at_index(4)
    lower = (rain_a.as_int() ^ 0xff)
    rain_b = input_data.get_byte_at_index(6)
    higher = rain_b.as_int() ^ 0xff

    value = (higher << 8) + lower

    value = value + 10
    value = value & 0xffff

    return value

I will make a PR with this. Now I need to figure out two more things: the bit for low battery and the parity/checksum. Basically, I have 5 bytes out of 8.5 figured out by now.

@zuckschwerdt Given that I have to invert bytes 4 and 6, it is reasonable to assume that I have to invert the entire stream?

edit:

Looks like the "low battery" bit is bit 24, i.e. first bit after the ID: bitbench

[mhaas]

@Fietspomp86 Can you try this code? https://github.com/merbanan/rtl_433/pull/1255

[Fietspomp86]

@Fietspomp86 Can you try this code? #1255

Hmm, it seems my rainmeter has died on me. It won't show any figures, also not on the display.... Guess I'll send it back to the shop. Sorry guys!

[mhaas]

@Fietspomp86 Maybe you can also try to reset the batteries - maybe the devices lost sync.

[mhaas]

I did some work on the checksum today

I noticed some interesting behavior: the sensor will repeatedly submit identical data with increasing transmission counter bits. Once the transmission counter rolls over, byte 7 is identical to previous transmissions with identical rain counter and transmission counter bits.

I assume now that byte 7 is indeed the checksum.

I already tried several approaches: reveng, xor of values, sum of values, two's complement etc. Nothing worked. I also tried leaving out the first nibble, as it is a fixed for all device IDs - no luck.

I should note that the checksum is different for identical rain and transmission counter values across different device IDs (e.g. after battery reset).

One interesting thing: the last bit of byte 7 determines the value of the first bit of byte 8: they will be identical.

I have prepared several bitbenches with different device ids:

• 1
• 2
• 3

@zuckschwerdt Any idea how I can proceed with the checksum?

[zuckschwerdt]

Looks very thorough, so nothing obvious to recommend. Not sure if revdgst (see https://triq.org/) can help. But it could be useful to quickly try some checksum ideas. Putting all of your BitBench data into a txt file and using revsum (or revdgst) shows somethings like nibble-wide parity. But that really needs some poking around to make sense of.

[zuckschwerdt]

One thing that might be visible in your sorted examples is too little true randomness in the checksum. It feels more like a sum (plain add with carry or xor without carry, or even some Fletcher-style sums).

[zuckschwerdt]

I do have to double check, but could indeed be a LFSR digest: reverse Galois with byte reflection, generator 31, key f4, final XOR c0).

[mhaas]

Sounds very specific already, I will be happy to do the double checking for you tomorrow :)

Am 11. Januar 2020 20:03:00 MEZ schrieb "Christian W. Zuckschwerdt" notifications@github.com:

I do have to double check, but could indeed be a LFSR digest: reverse Galois with byte reflection, generator 31, key f4, final XOR c0).

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/merbanan/rtl_433/issues/1240#issuecomment-573344944

-- Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

[zuckschwerdt]

Interesting thing to note: LFSR digests are somewhat weaker checksums than CRC, so no reason to bother really. But a Galois digest might very well be a native feature of the used chips. So maybe the designers just went with that because it's easy.

It might look like a nice security feature to novice designers but 8 bit keystreams are toys from a serious security standpoint. Even 16 bit LFSR are not too hard to crack -- revdgst was made for that and the 8 bit brute-force came with that because it takes no computational power whatsoever ;)

Harder to make sense of are the botched CRC implementations in some protocols -- though when you finally see where the designers went wrong it's a great laugh :)

[zuckschwerdt]

You can get the keystream with keylst -g 31 -i f4 -G -r -w 8 from revdgst. Since messages are 64 bit we need 64 keys: f4 d9 83 37 6e dc 89 23 46 8c 29 52 a4 79 f2 d5 9b 07 0e 1c 38 70 e0 f1 d3 97 1f 3e 7c f8 c1 b3 57 ae 6d da 85 3b 76 ec e9 e3 f7 df 8f 2f 5e bc 49 92 15 2a 54 a8 61 c2 b5 5b b6 5d ba 45 8a 25 Byte reflection then means the keys apply from last byte to first byte (high bit to low bit, i.e. left to right). Of course this means you can shuffle the keys to go first byte to last byte. And if you also shuffle to go low bit to high bit the progression could perhaps be continuous, i.e. you could create a generator that's more plausible. Best to stuff the keys in a BitBench and follow the shift pattern.

[mhaas]

@zuckschwerdt Now you have me a bit confused. My goal was to understand the computation of the checksum so that I can re-implement it. With my own implementation, I can compute a checksum from the message and compare with the transmitted checksum. Messages where the checksum does not match are discarded.

Above, you provided the output of your revdgst tool:

reverse Galois with byte reflection, generator 31, key f4, final XOR c0).

With this information, I was able to re-implement (or rather, copy&paste from your code :)) the checksum algorithm and it seems to work: https://github.com/merbanan/rtl_433/pull/1255/files#diff-0391fbab6a3dbb6c0ddbc0e5a1e7d82aR226

The code is rather heavily commented as I was trying to understand what was happening there :)

Is there anything else I am missing here? What could I get from revdgst that I am missing? Or where you simply giving me some pointers how I can validate the parameters given by revdgst?

I have two more questions related to revdgst - I am really ignorant on this stuff.

• What is the meaning of the final XOR c0? In my code, I do not need a final XOR.
• In your runner() function, you first compute the digest/checksum of the first data line - and then the checksums of the remaining lines, while comparing these checksums with various operations to the initial checksum from the first line. What does it do? I did not seem to need that information at all.

This is now the second time where you just produced a tool that really solves my problem. Next time, I will ask directly :)

The PR in #1255 should now be ready for a review!

[zuckschwerdt]

Oh, sorry, just rambling there. That algorithm is just one example which works in producing a stream of keys to use for each bit. Each bit in the protocol has a fixed key (the list above taken from keylst) and those keys are xored for all set bits. There is no given order of keys just convenient orders where a simple LFSR can generate the keys. And because "reverse" and "byte reflect" really look wrong I wanted to point out that perhaps another key order also has some generator, which could look more sanely.

The final XOR (c0) should be the fixed point that all valid codes result in. Usually we'd want a zero there but that's arbitrary. Perhaps there is a bug in revdgst and indeed it's zero here, like your code shows.

The runner just tries a number of methods how the checksum could be composited and stored. E.g. each step of the checksum could be xored/added/subtracted/... and the result stored as the directly the final value/addition/substraction/... remainder. In this case it's xor, xor.

[merbanan]

@mhaas @zuckschwerdt can this issue be closed now?

[Kwenk]

@Fietspomp86 Can you try this code? #1255

Hmm, it seems my rainmeter has died on me. It won't show any figures, also not on the display.... Guess I'll send it back to the shop. Sorry guys!

I bought this sensor because of the support here, but despite the update I dont get any data. Its a very rare sensor. Is there any feedback that it works correctly? The initiator probably never tested it himself?!

[mhaas]

Can you get sample data as described above?

[Kwenk]

Of course, here are my samples: kwenk_raindrop.zip

[Kwenk]

Not ok? Anything else I could do?

[zuckschwerdt]

Looks like your samples are "too loud", the center frequency (DC) might be too close to your signal. It might help to offset e.g. 50kHz: -f 433.97M

[Kwenk]

Nice hint with 433.97M - I used 433.92M - worked with my different kind of temperature sensors but not with this rain sensor. With -f 433.97M I got it every 45 seconds:

model     : TFA-Drop     id        : d93f5
Battery OK: 1            Rain in MM: 4.826         Integrity : CHECKSUM

I'm picking up my other sensors as well. Sorry for the inconvenience.