New TFA 14.1504.V2 sensor not supported by any decoders

joel-bourquard commented 1 year ago

EDIT: Turns out that this sensor is a V2 and due to that, has actually nothing to do with the Maverick, so it will likely require a new decoder. So the first few posts contain some erroneous assumptions.

I bought a new TFA 14.1504 BBQ sensor that reportedly uses the same protocol as "[71] Maverick ET-732/733 BBQ Sensor", according to this discussion: https://github.com/RFD-FHEM/RFFHEM/issues/167

The TFA manual that comes with my device confirms that it's 433Mhz, and I have several other 433Mhz sensors working with rtl_433 already.

I built rtl_433 from HEAD (master). Yet, nothing gets picked up from the TFA.

I tried some fiddling such as: ./rtl_433-s 1024k -Y autolevel -Y magest -M noise -M level -Y minmax

which picks up more faraway sensors from my neighborhood, but nothing from the TFA that is a few meters from my antenna.

The discussion above also links to a commit in a different project that adds support from TFA 14.1504 to their Maverick ET-732/733 decoder, maybe that could give us very good hints? https://github.com/RFD-FHEM/RFFHEM/commit/b50c779be083dded778397f55238eb0500091f15

In rtl_433, I tried adding some logs to maverick_et73x.c:

static int maverick_et73x_callback(r_device *decoder, bitbuffer_t *bitbuffer)
{
    data_t *data;
    bitbuffer_t mc = {0};

    if (bitbuffer->num_rows != 1)
        return DECODE_ABORT_EARLY;

    //check correct data length
    if (bitbuffer->bits_per_row[0] != 104) // 104 raw half-bits, 52 bits payload
        return DECODE_ABORT_LENGTH;

    //check for correct preamble (0x55666a)

From what I saw, the data length is never equal to 104 and thus the function exits with DECODE_ABORT_LENGTH. I don't understand how it can abort so early, if the protocol is really similar to Maverick ET-73x.

I would be happy to help but I don't know what to do.

zuckschwerdt commented 1 year ago

Can you grab a sample and upload as zip here? https://triq.org/rtl_433/ANALYZE.html

Your debugging is spot on. Use bitbuffer_print(bitbuffer); (or bitbuffer_debug) or e.g. even decoder_log_bitbuffer(decoder, 0, __func__, bitbuffer, "before lenght check") to get some insight.

joel-bourquard commented 1 year ago

Thank you! In the meantime I tried rtl_433's pulse analyzer, focusing on the results that have best SNR (ie: around 20 dB). After some playing around, I came up with this command: ./rtl_433 -R 0 -X 'n=name,m=OOK_PWM,s=10000,l=15000,r=70660,t=1414,y=0'

It's not 100% accurate, but most of the time, it catches 1 signal every 5 seconds -- which corresponds to the TFA's transmit interval:

rtl_433 version 22.11-26-g616b5d5e branch master at 202212201349 inputs file rtl_tcp RTL-SDR Use -h for usage help and see https://triq.org/ for documentation. Trying conf file at "rtl_433.conf"... Trying conf file at "/root/.config/rtl_433/rtl_433.conf"... Trying conf file at "/usr/local/etc/rtl_433/rtl_433.conf"... Trying conf file at "/etc/rtl_433/rtl_433.conf"... Disabling all device decoders. Protocols: Registered 1 out of 223 device decoding protocols [ ] SDR: Found 1 device(s) SDR: trying device 0: Nooelec, NESDR SMArt v5, SN: 386b0ef3 Found Rafael Micro R820T tuner SDR: Using device 0: Generic RTL2832U OEM Exact sample rate is: 250000.000414 Hz [R82XX] PLL not locked! SDR: Sample rate set to 250000 S/s. SDR: Tuner gain set to Auto. SDR: Tuned to 433.920MHz. Allocating 15 zero-copy buffers

time : 2022-12-23 12:23:23 model : name count : 5 num_rows : 5 rows : len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : codes : {0}, {0}, {0}, {0}, {0}

time : 2022-12-23 12:23:28 model : name count : 5 num_rows : 5 rows : len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : codes : {0}, {0}, {0}, {0}, {0}

time : 2022-12-23 12:23:33 model : name count : 5 num_rows : 5 rows : len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : codes : {0}, {0}, {0}, {0}, {0}

time : 2022-12-23 12:23:38 model : name count : 5 num_rows : 5 rows : len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : codes : {0}, {0}, {0}, {0}, {0}

time : 2022-12-23 12:23:43 model : name count : 5 num_rows : 5 rows : len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : codes : {0}, {0}, {0}, {0}, {0}

time : 2022-12-23 12:23:48 model : name count : 1 num_rows : 1 rows : len : 2 data : c codes : {2}c

time : 2022-12-23 12:23:48 model : name count : 5 num_rows : 5 rows : len : 0 data : , len : 0 data : , len : 1 data : 8, len : 0 data : , len : 0 data : codes : {0}, {0}, {1}8, {0}, {0}

time : 2022-12-23 12:23:53 model : name count : 5 num_rows : 5 rows : len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : codes : {0}, {0}, {0}, {0}, {0}

time : 2022-12-23 12:23:58 model : name count : 5 num_rows : 5 rows : len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : codes : {0}, {0}, {0}, {0}, {0}

time : 2022-12-23 12:24:03 model : name count : 5 num_rows : 5 rows : len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : codes : {0}, {0}, {0}, {0}, {0}

time : 2022-12-23 12:24:05 model : name count : 1 num_rows : 1 rows : len : 1 data : 8 codes : {1}8

time : 2022-12-23 12:24:08 model : name count : 5 num_rows : 5 rows : len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : codes : {0}, {0}, {0}, {0}, {0}

time : 2022-12-23 12:24:13 model : name count : 5 num_rows : 5 rows : len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : , len : 0 data : codes : {0}, {0}, {0}, {0}, {0}

So this is encouraging. Edit: And when I power off the thermometer, these messages don't come -- so these messages are definitely it!

I've tried some .cu8 captures, but rtl_433 seems to generate a bunch of very short files. How could I, say, have a single file that contains 20 seconds of data? That way I'd be sure that it would contain at least 3 full signals in it...

joel-bourquard commented 1 year ago

Here's a full 60-second capture taken with this command: rtl_433 -s 1024k -Y autolevel -Y magest -M noise -M level -Y minmax -S all -T 60 https://matrixrewriter.com/tmp/capture1.zip The interesting transmission occurs every 5 seconds, but I don't know how to isolate it...

joel-bourquard commented 1 year ago

And here's the results of a run with the "before length check" code added, as follows:

if (bitbuffer->num_rows != 1)
    return DECODE_ABORT_EARLY;

decoder_log_bitbuffer(decoder, 0, __func__, bitbuffer, "before lenght check");

//check correct data length
if (bitbuffer->bits_per_row[0] != 104) // 104 raw half-bits, 52 bits payload
    return DECODE_ABORT_LENGTH;

and the command was: ./rtl_433 -R 71 &> before-length-check.txt

Here's a 60-second extract: before-length-check.txt (I see lengths like 127, 129, 131 but I don't know if they come from this device or not)

joel-bourquard commented 1 year ago

And here's a 60-second capture of demodulated pulses obtained with this command: ./rtl_433 -R 0 -w OOK:- &> demod.txt demod.txt

I don't fully understand the contents, but we should look at everything that is around 20dB SNR and comes every ~ 5 seconds.

joel-bourquard commented 1 year ago

And here's a 60-second analysis by rtl_433 with 1024k sample rate, obtained with this command: ./rtl_433 -R 0 -s 1024k -A &> analysis.txt Results: analysis.txt

zuckschwerdt commented 1 year ago

That should be a good amount of data to analyze this.

To answer a few questions: Get a fixed time capture using -w, e.g. rtl_433 -w myfile_433.92M_250k.cu8 -T 60

OOK files are also great (and compact), to see the results use -w OOK:- but to save a file use -w demod.ook (or -w OOK:demod.txt

zuckschwerdt commented 1 year ago

Oh, there are at least three different transmission types in your recordings. None look like the ET-73x (OOK, Manchester, half-bit width of 230 µs).

You need to try and isolate the signal, too much guesswork otherwise.

joel-bourquard commented 1 year ago

Thanks for your feedback and analysis! Yes, this model might be different from the ET-73x, who knows... Here's something that might help narrow it down: in the "analysis.txt" file, search for "46474". I am convinced that this value is tied to the device, as it reappears every 5 seconds!

zuckschwerdt commented 1 year ago

That "46474 us" is just a FSK signal that's mistaken for OOK. You can drop those files on https://triq.org/pdv/ to visually compare. They look very distinct. If you can indeed confirm that transmission, then this is a very different protocol: FSK not OOK, and PCM not Manchester.

franki29 commented 1 year ago

HI , first of all merry Christmas to all of you and your families.

@joel-bourquard

It would be nice if you can take off the antenna of your SDR dongle and than capture the signal of your TFA BBQ Sensor with the current temperature that are shown at your device.

You are getting so much different sensor data and the level are too high I think.

If you look at your files with *rtl_433 -f 433.95M -r .cu8 -Y level=-5**

You get data from a:

time : @0.377813s model : AmbientWeather-WH31E id : 210 Channel : 1 Battery : 1 Temperature: 19.3 C Humidity : 73 % Extra Data: 736c000200 Integrity : CRC time : @0.252629s

model : Fineoffset-WN34 ID : 2fb9 Battery : 0.2 Battery Voltage: 1220 mV Temperature: 21.9 C Integrity : CRC

With *rtl_433 -f 433.95M -r .cu8 -Y level=-20**

time : @0.431235s model : Regency-Remote channel : 1 command : light_intensity value : 250 % mic : CHECKSUM

joel-bourquard commented 1 year ago

Hi all, and thanks for your feedback and help! This is great :-)

Time was very tight today but I removed the SDR antenna, connected it to a different computer (that has an older rtl_433 version 21.12) and put the TFA 14.1504 just in front of the SDR.

Here's a new analysis done with this command: rtl_433 -R 0 -s 1024k -A &> analysis2.txt analysis2.txt

Much better: this new analysis only shows the TFA device itself, every 5 seconds, which is good!

And here's a new capture done with the command below, while the thermometer indicated 20 degrees Celsius: rtl_433 -s 1024k -Y autolevel -Y magest -M noise -M level -Y minmax -S all -T 60 capture2.zip

Ah ah, now the files are much smaller, hopefully thanks to the lower interference levels. Only downside is that the two FineOffset thermometers are not very accessible so I could not disable them. So even without an antenna, they popped up due to the increased sensitivity. But they only broadcast once a minute or so, while the TFA does it every 5 seconds -- so hopefully this will be sufficient.

Merry Christmas to you all! Take care :-)

joel-bourquard commented 1 year ago

Hi @franki29, @zuckschwerdt, Hope you had a Merry Christmas :-)

This is a WIP, but I wanted to share the progress. I successfully isolated and identified the TFA 14.1504 signal: signal Here's the CU8 file (remove the .pdf extension): g130_433.92M_1024k.cu8.pdf

and here's my first successful attempt to digitize it: ./rtl_433 -R 0 -X 'n=name,m=FSK_PWM,s=1000,l=2000,r=8000,invert'

I've also successfully added this: preamble={12}800

On the following points, some advice may help:

Is this really FSK_PWM? And how about the flex parameters that I use? I have doubts as the resulting number of bits seems to vary slightly and the binary sequences are a bit weird.
As seen on the right of the spectrogram, the signal is repeated, but I don't know if/how I should get rtl_433 to identify them as multiple rows or repetitions, in order to maximize stability. Suggestions welcome!

For now, every 5 seconds I get the same reading 2, 3 or 4 times eg: time : 2022-12-28 07:22:52 model : name count : 1 num_rows : 1 rows : len : 13 data : 4100 codes : {13}4100

and here's a (wildly incomplete) conversion table, from degrees Celsius to hex code: 6 or 7 = 800428 8 = 8004e0 9 = 8004a8 10 = 8004c0 11 = 8004d2 12 = 800448 13 = 800428 15 = 800432 16 = 800424 17 = 800415 20 = 800430 22 = 800410 23 = 80041c 24 = 800424 25 = 800470 ? 27 = 80046c 28 = 800460 29 = 800422

Note: the table includes the "0x800" preamble and the hex values are weird, but at least I get a value that correlates with a temperature reading...

franki29 commented 1 year ago

Hi @joel-bourquard , thanks, yes I had a great Christmas days. Regarding your captures, I am not sure if you really used the right parameters. Looking with audacity at the signals, they look very strange to me. None of them looks like the Maverick files you find in the issues from other Maverick related issues. I am not sure what parameters you should use to capture signal.

I tried with your capture2 files rtl_433 -r g001_433.92M_1024k.cu8 -Y minmax -X 'n=name,m=FSK_PCM,s=360,l=360,r=4096' and got some data.

Maybe you can try rtl_433 Y minmax -X 'n=TFA-Test,m=FSK_PCM,s=360,l=360,r=4096' and look if you get consistent data.

grafik

zuckschwerdt commented 1 year ago

@franki29 they look that way because the FSK is crossing DC (0 Hz), so both mark and space frequency "look" the same -- you only notice the DC crossing. Also it's GFSK, a slow change in frequency not a "jump".

joel-bourquard commented 1 year ago

Hi @franki29, thanks a lot! I have updated the title of this ticket, as this TFA device turns out to be very different from the Maverick.

Thanks a lot for the new parameters! With short/long set to 360, the number of bits was 85-86 (not 100% constant). I found a sweet spot from 380 to 410 that gives a consistent 85 bits, so I changed short/long to 400 for now.

At a temperature that is (I think) 28 degrees Celsius, I get either of these sequences: {85}3d555552e14607f81f6000 {85}3d555552e14607f81f6030

I'll do some testing at different temperatures tonight, but at least the data looks stable and contains several 0x55 bytes which is a good sign! and there would be enough room for a CRC or similar. I'll keep you updated on what I find.

zuckschwerdt commented 1 year ago

Add -vvv -R 0 to see the PCM slicer locking on to the actual bitrate found in the 55…, e.g. rtl_433 -vvv -R 0 -Y minmax -X 'n=name,m=FSK_PCM,s=360,l=360,r=4096' g001_433.92M_1024k.cu8

[pulse_slicer_pcm] General purpose decoder 'name': Exact bit width (in us) is 367.97 vs 359.38, 30 bit measured

joel-bourquard commented 1 year ago

Ah, very interesting! But when running your command either from 22.11 or latest master, I don't get this log line from pulse_slicer_pcm. Did you use a special version of rtl_433? Or does it come from the rtl_sdr library?

zuckschwerdt commented 1 year ago

That output has been there since 4282e4e6, included in 20.02 release. But you need -vvv (-R 0 just to supress the other protocols).

joel-bourquard commented 1 year ago

Hi @zuckschwerdt, Ah, strange that I didn't see it - I'll try it again eventually.

I have very good news: I successfully reverse engineered the checksum algorithm and parameters, so I think I will implement this in code, and the make a PR to add this protocol to rtl_433 😊

Edit: Aargh, for the checksum I spoke a bit too soon - but I will continue regardless!

joel-bourquard commented 1 year ago

Hi @zuckschwerdt, The CRC reversing was super tough, I had to use two different kinds of CRCs plus a look-up table, but it works!! It took a lot of effort. Maybe a wizard can simplify it in the future.

Therefore, I'm proud to submit my pull request that adds support for this device to rtl_433: https://github.com/merbanan/rtl_433/pull/2296

A couple notes/questions:

it reports temperature as an int (not float), is it a good idea?
if the probe is unplugged it refrains from reporting anything. Is there a better way to report "no value" for the temperature? In that case I could also add another attribute named "Probe status" which would be neat.
I have testcases to validate the CRC computations, I could contribute them in the other project eventually.

zuckschwerdt commented 1 year ago

Thanks, looks good.

Most outputs should be okay with int vs float, but I'd just output a float anyway since it might be what consumers of the data expect. You can omit a key with DATA_COND -- used a few times in other decoders.

joel-bourquard commented 1 year ago

Thanks, looks good.

Most outputs should be okay with int vs float, but I'd just output a float anyway since it might be what consumers of the data expect. You can omit a key with DATA_COND -- used a few times in other decoders.

Awesome! I've changed it to float, added probe status reporting and made the temperature optional based on it. Here are some screenshots of the results: Screenshot_20230102-193425__01__01 Screenshot_20230102-193326__01

joel-bourquard commented 1 year ago

Closing this ticket, as everything is implemented in my PR: https://github.com/merbanan/rtl_433/pull/2296

merbanan / rtl_433

New TFA 14.1504.V2 sensor not supported by any decoders #2288

EDIT: Turns out that this sensor is a V2 and due to that, has actually nothing to do with the Maverick, so it will likely require a new decoder. So the first few posts contain some erroneous assumptions.