Support for Watchman Sonic Advanced/Plus

[garethpotter]

Yet another Watchman oil tank monitor, with an apparently different format.

A few samples are at https://github.com/merbanan/rtl_433_tests/pull/440.

I have managed to decode these using URH following the process outlined in https://github.com/merbanan/rtl_433/issues/1333. The signal is clear enough in URH's demodulated view:

...but I am not sure how to translate that process into a rtl_433 flex decoder. Running the captures through rtl_433 -A -a 4 doesn't yield much of use, at least from what I can see.

[zuckschwerdt]

That is a pretty neat signal capture. You can even just drop the cu8 onto https://triq.org/pdv/ and see the same curve. It's GFSK, which rtl_433 isn't optimized for. But the minmax demod can pick that up somewhat, e.g. rtl_433 -Y minmax -A g100_433.92M_1024k.cu8 -- the timings are off though, actually it's 500 µs PCM like you manually found. E.g. should work, but does not: rtl_433 -Y minmax -X 'n=name,m=FSK_PCM,s=500,l=500,r=9000'

We really need someone to build a GFSK-optimized demod someday.

[garethpotter]

We really need someone to build a GFSK-optimized demod someday.

I suspect this is well beyond my capabilities but I will have a dig around.

[zuckschwerdt]

The code is here: https://github.com/merbanan/rtl_433/blob/master/src/pulse_detect_fsk.c#L158 I suspect that setting FSK_EST_SLOW to up to 10x the value might help: https://github.com/merbanan/rtl_433/blob/master/src/pulse_detect_fsk.c#L23

[rolandshiner]

I just got a Kingspan Watchman Sonic Advanced which I was hoping would work with a rtl-433, but like others here I find sadly it doesn't.

I'm pretty new to rtl-sdr etc but I've been sniffing the airways and have found some useful data. I think I have an ID and both temperature and height but there is other data here I haven't understood yet. The below very long link is my test data on bitbench. *I'm not sure how best to link to that.

The heights shown are always within 1-2 cm of the bottom of the device and the floor. I'm pretty confidant the height data is correct.

The temperature in the room I'm testing in is approx 18.5C, but it's possible it's reading a little low and the 15.4C shown may be because I put the device on the cold stone floor while adjusting the tripod I am hanging it from. I will try to test the temperature idea further.

BitBench data

I capture signals using 'rtl_433 -f 433.92M -S unknown' I use Universal Radio Hacker to analyse them with default settings except for: Center: -0.52 Samples/Symbols: 125 Signal View: Demodulated Show Signal as: Hex

I don't know what to do with it from here but maybe others with more experience can take a look.

[zuckschwerdt]

Thanks, the 2dd4 is a known sync word, payload starts after that, then the last 16 bits are a CRC-16 poly 0x8005 init 0.

[garethpotter]

Right, using my BitBench format string (40b 8h 8h 8h 8h 8h SERIAL: 24d 8h TEMP: 8b 8h 8b DEPTH: 8d 8h 8h 8h 8h CRC: 24h) I get a serial number of 05261860 for yours. I note the preamble before the serial number is the same for both devices, though the 8 bits after the serial number differ.

In my case, I used Wireshark to intercept the HTTP POST submissions the USB stick was making (using the firewall to block HTTPS to force it on to HTTP). As I note in the README in my rtl_433_tests submission, I am not quite sure how the temperature is encoded, though I know from the HTTP request that it is just a single 8-bit integer (and negative temperatures overflow).

[rolandshiner]

Serial number on the device is '5261860'. Clearly they dropped a preceding '0'.

[zuckschwerdt]

Combined and clean up the format string is PRE:40h SYNC:16h MODEL?24h ID:24d ?8h ?8h TEMP:8d ?8h DEPTH-CM:8d 8h 8h 8h 8h CRC:16h

[rolandshiner]

I'm becoming less convinced by my temperature assumption. I'm sometimes seeing it effectively be 026 (2.6C) or 058 (5.8C) while still in the same room at 18.5C.

I'm printing a better tripod mount and then I'll get back to testing

Other than temperature and depth what other information are we expecting to find within the data part of the signal?

[merbanan]

Whatever the other watchmen sensors have. But battery status should be somewhere.

[garethpotter]

Based on the HTTP submissions, little else. I suspect the temperature is encoded in some way spread across the 8h before the TEMP:8d and potentially afterwards as well, because in my recordings, that 8d does not change (it is always 170).

To elaborate, using the following partial format string, SERIAL: 24d 8h TEMP: 8b 8b 8b DEPTH: 8d:

TEMP: 01001001 10101010 01010000 is 0ºC TEMP: 01001011 10101010 01100000 is 1ºC TEMP: 01001101 10101010 01100000 is 2ºC TEMP: 01001111 10101010 01100000 is 3ºC

When I did these recordings it was too cold outside to get higher temperatures! It did actually hit 10º today but I wasn't capturing transmissions.

[garethpotter]

There is one other seldom-changing value in the HTTP data. I don't think it is the battery though: it has ranged between 20 and 27 since I started capturing that data at the start of December.

[zuckschwerdt]

Other common flags/values include binding, counter, leak/theft alarm https://github.com/merbanan/rtl_433/blob/master/src/devices/oil_standard.c#L64 https://github.com/merbanan/rtl_433/blob/master/src/devices/oil_watchman.c#L63

[rolandshiner]

I can confirm what I had previously thought might be temperature is definitely NOT temperature. I've now got records for 18C, 11C & 8C (all approx)

The unknown 4 bytes between ID and DEPTH in binary are: 11000000 01101000 10101010 01110000 at 18C 11011000 01100001 10101001 01100000 at 11C 11011000 01010100 10101001 01100000 at 8C

@garethpotter Can you see temperature in the app? I don't have the USB receiver.

The basic plug in receiver I have does show low battery so there must be either a voltage level or low battery flag.

I'm pretty sure low level and empty are calculated on the receiver and the device knows nothing about that. It just reports depth in that regard.

Sudden drop is fairly simple to calculate in the receiver but could possibly be checked on the device. I'll try to test for that at some point.

I'll rebind my receiver at some point to look for that data.

I tested for 'NO ECHO CONDITION' but there doesn't appear to be a flag relating to that in the data that I could see but instead I think the warning is just based on chaotic data being received.

[garethpotter]

@rolandshiner As far as I know, the temperature is not visible anywhere. I am extracting it as detailed above and dumping it into Home Assistant so I have it there.

I am hoping the temperature will drop below zero over the next 24 hours, so with a bit of luck, I should have some more interesting readings soon.

[rolandshiner]

I've been capturing regular, every 30 minute, signals over night to test with rtl_433 using @zuckschwerdt suggested configuration of rtl_433 -Y minmax -X 'n=name,m=FSK_PCM,s=500,l=500,r=9000'. The results are hit and miss. Sometimes it's perfect, sometimes the first part of the data result is correct but presumably an extra detected 1 or 0 then throws off the rest of the data. There are also cases where the data would be correct but it missed the first 1 of the preamble so the data gets distorted accordingly.

I've attached a zip of my test files from overnight. If anyone can use them to further enhance reading the data that would be great, sadly it's well beyond my current understanding.

watchman advanced test files.zip

[garethpotter]

OK, your captures are a bit more useful than mine, presumably because your antenna is closer to the transmitter. I have knocked together a very rudimentary decoder that manages to parse out the serial number and the depth.

https://github.com/garethpotter/rtl_433/commit/0fdd337074d91037fcc3f0140a11529bac1fd03c

It seems to work on most/all of your sample files.

However, I'm still struggling to understand the relationship between the temperature seen in the HTTP POST data and that on the wire. I've had two new values today:

45 aa 50, which corresponds to 255, (i.e. -1º) 43 a9 50, 254, (i.e. -2º)

...but I am still stumped.

[zuckschwerdt]

So far we have this list?

43a950 [-2 C]
45aa50 [-1 C]
49aa50 [0 C]
4baa60 [1 C]
4daa60 [2 C]
4faa60 [3 C]
54a960 [8 C]
61a960 [11 C]
68aa70 [18 C]

A linear regression on the first byte yields temp C = 0.5205 * Byte - 37.48 Extreme temperatures would be needed to confirm this and get better coefficients.

[merbanan]

It could be raw sensor values and then we need a lookup table and/or piece-wise interpolation. I've seen this one time before. We need a lot more readings to figure this out.

[rolandshiner]

Allowing for our seperate recorded temperatures to be off by a degree or two in comparison to the temperatures the transmitter recorded: b / 2 - 35

[merbanan]

This is linear with an offset of 0x46.

[rolandshiner]

Exactly. Nice and simple.

[merbanan]

I think we can let @garethpotter complete his decoder with this. It looked fairly ok from a brief look.

[garethpotter]

I've tweaked the temperature calculation accordingly. Not had any luck calculating the CRC yet. This is the first time I have written any C code, or indeed contributed to an open source project, so please calibrate your expectations accordingly.

A few more temperatures to round out the data:

3c aa 40 maps to 250 3e a9 40 maps to 251

[garethpotter]

And one more overnight:

3a aa 40 is 249

[rolandshiner]

@garethpotter so using that data I realise 0x45 is 0C and not 0x46.

In summary: The USB receiver formatting is a signed byte in whole degrees C. The transmitter formatting is half degrees C with an 0x45 offset.

[zuckschwerdt]

Not had any luck calculating the CRC yet.

The last 16 bits are a CRC-16 poly 0x8005 init 0. The check would be something like

        if (crc16(b, 17, 0x8005, 0) != 0)
            return DECODE_FAIL_MIC;

[garethpotter]

Right, this is done now. I've submitted a PR (https://github.com/merbanan/rtl_433/pull/2323).

[garethpotter]

@rolandshiner Should I add some of your samples to the test repository or do you want to do that? They are more useful than mine as they can actually be decoded.

[rolandshiner]

@garethpotter Yes, feel free to do that.

[garethpotter]

Right, that's done.

[Mobbsey]

Hi,

I've updated rtl_433 to include the changes in #2323 (which btw, thank you for this amazing contribution!), but I don't get any readings coming from my Watchman Sonic Advanced... If I run rtl_433 with -vv, I can see the following screenshot. It seems that it is selecting the correct decoder, but never manages to decode the message.

The full command I used to get the screenshot below is: rtl_433 -vv -R 234 -Y minmax

I'm no expert on rtl_433, but wondered if there was anything obvious I'm missing? Any help would be apprecaited.

[zuckschwerdt]

Can you grab more of those codes and copy/paste as text? It looks like bad reception though.

[zuckschwerdt]

Can you grab more of those codes and copy/paste as text? It looks like bad reception though.

[garethpotter]

What I’ve found is that if the signal is too strong, it doesn’t work. At the moment, I’m using a gain setting of 36.4 with the antenna inside a shed about 5m from the transmitter. That is decoding reliably every 30 minutes.

If you still don’t have any luck, I suggest trying to capture the signals with -S unknown (best to do that with all the default encoders enabled, otherwise you’ll end up with a load of unknown knowns, as Dick Cheney might have it) and then loading them in to URH. That at least would give you a sequence of bits to put in BitBench so we can see if any of the things we thought were constant actually turn out not to be.

[garethpotter]

Would also be good to know the serial number of your device so we know when we've found it.

[Mobbsey]

Thanks for the quick response both. Let me do some more captures over the next couple of days and I'll come back. Interesting re the signal strength - it's indeed sat on the desk at the moment whilst I tested, so I'll resite it back outside and see if that makes a difference. Thanks so much - I'll feedback what I find FYI.

[Mobbsey]

Gents, thanks again for your help. I moved the transmitter back outside, and as per your comment @garethpotter, it's worked reliably overnight. Might be worth a note in the README re the signal strength?

Thanks again, hugely appreciated.

[garethpotter]

Hurrah.

I can at least add something to the header of the decoder source file. I might need to make a tweak to the temperature calculation as well - am waiting for the weather to warm up to see how accurate it is - so if so, I can submit a PR for both in due course.

[Mobbsey]

If you want a second source of logs to validate temperatures etc once the weather's warmed a little, feel free to reach out. Happy to help.

[arfond]

Hi. Would this by any chance work for my plus device? Info on it here: https://github.com/merbanan/rtl_433/issues/2133

[zuckschwerdt]

Hi. Would this by any chance work for my plus device? Info on it here: #2133

No, your device codes with variable width symbols and bit stuffing vs. this "CC1101" standard coding.

[wonkosoft]

Hi. Firstly - thanks for the work people have put into this. It was perfect timing for me - almost exactly coinciding with when I decided I was going to try and set up monitoring of my two oil tanks using a couple of these devices. As a software engineer I'd have been happy writing this code, but I'm completely new to decoding radio signals, so having someone else do that hard work was awesome.

Okay. I've got a couple of observations that can hopefully help to further the work, that I'll drop below.

[wonkosoft]

First observation - the byte directly preceding the temperature (position 7, using 1-based indexing). In my experiments, that has only ever taken 3 distinct values : 0xC0 - this is during the first 20ish minutes when you've synced the configuration with the receiver and the device is transmitting once per second. 0x80 - this is the first one or two transmissions after the sync period when the device seems to be calibrating itself. 0x98 - this is its normal, live value that you'll see on every transmission when the device is up and running

I'm happy to add that as a flag to the output, but thought it would first be worth seeing if anyone had seen any other values here.

[wonkosoft]

Second observation - I had my tanks filled the other day. I only had the one device installed at that point. For a period of a day or two afterwards I was unable to get a reading through RTL_433, as if the device had stopped transmitting. Remembering that I'm very new to the radio side of this, I did my best to debug the problem. I found that the device was definitely still transmitting. I thought there might be a problem with noise, but I'm not at all convinced by that because I did see very clear signals and, it seemed, the code was trying to decode that but failed.

I came down to it being one of two possible causes, but couldn't investigate further as by the time I got to this point the problem essentially sorted itself out. The first possibility was that (because of the fill, which was very high in the tank) the device went into its "No echo" mode and that the transmission in that mode is not of the same length as a regular transmission. The second possibility, that seems to be more likely, is that there is quite a long part of the transmission which is 'low' (so, maybe a collection of zeros) and the demodulator has incorrectly determined that the transmission has ended, and then a new transmission has started.

Edit: forgot to mention that the supplied receiver was showing its "no echo" display at this time.

I've attached two signal snippets in .cu8 g002_433.92M_250k.cu8 is able to be decoded. g004_433.92M_250k.cu8 is not able to be decoded.

Wondered if someone who had a better understanding of this stuff can indicate why I can read one but not the other.

samples.zip

[wonkosoft]

Third observation - there's two bytes between the temperature and depth (so positions 9 and 10 using 1-based indexing). I'm convinced that this is either (1) some representation of time taken for the sonar echo to be received or (2) some representation of strength of sonar echo received, one of which would be used in conjunction with the temperature (as speed of sound changes by temp) for the device to calculate distance. I feel it's more likely (2) as, for a period when the tank was very full I was getting a lot of erroneous readings (all over the place) and this value was always inversely proportional to the reported distance. I think basically the fullness of the tank was causing the device to incorrectly use a secondary echo to measure the distance.

[wonkosoft]

Last observation - the constant bytes at positions (1-based) 12,13 and 14 (possibly incorporating 15, which is always zero) look exactly like some kind of version number - either hardware or software. Mine would always read 1.5.3.0.

[zuckschwerdt]

Thanks, great infos. For the undecoded signal. Run with -M protocol on a good signal. You see the protocol number is 234, now switch that protocol to verbose with -R 234:vv and run also on the bad signal. You'll observe the code being chopped off like you guessed. There are 19 sequential 0's, so 9500µs gap, which exceeds the 9000 in the decoder. Change the decoder to .reset_limit = 12500, // allow 24 sequential 0-bit's and it will work. I'll update that.

[zuckschwerdt]

And there is likely a fourth status byte for low battery. I suspect it's a bit field and battery-good might be the first bit 0x80.

[wonkosoft]

Blimey. That was a fast response !!

Thanks. I didn't realise you could do that (-R 234:vv) on the protocol switch - that's very useful. And great news on the solution. I hadn't understood that part of the decoder config, and assumed the problem was earlier in the pipeline, i.e. in the initial demodulation of the FSK signal.

Edit: Interesting to see that this gives a depth of 0 in that reading, which I guess is the no sonar value - in fact the distance should have been 3 or 4 cm, but I guess the sonar doesn't work at that small a value. Certainly for 24/36 hours afterwards the readings were just all over the place.

And ... ah. Battery. Yes ... before I started exploring the idea of the values at positions 9 & 10 being related to time/power of the sonar, I did wonder whether the second byte (or, in fact, the 3rd nibble as the 4th is always 0) was related to battery power output. But it was very much a "clutching at straws" theory - it was just because my values for the second byte were always either 0x70 or 0x80, and it seems that it's always the lower value in lower temperatures, which could affect battery output. But, yeah .... probably not. Though I'll watch that value over time (I've added those flags to my local code to see what happens to them).