Hack hideki.c for receiving T/H on Bresser 5ch (7009993)

[robertbiloute]

I have a Thermometer/Hygrometer sensor from Bresser ("Bresser 5ch", id: 7009993), and rtl_433 didn't manage to read it properly. I investigated a bit, asked Bresser which protocol they use, they directed me to the hideki TS04 protocols, which I found in hideki.c.

Investigating further, I saw this protocol 42 was not firing any analysis of pulses because the parity bits were already wrong. I have 89 bits incoming (actually 3 packets of ~78, 89 and 89 bits for each transmission, the last 2 are repeating) instead of the 90 bits I shoud get (10 octets + 10 parity bits). I finally remarked that shifting the whole 89 word to the right by 2 bits, I got the right parity bits and data payload. Why is that ? I don't know wether it's a faulty sensor or a custom implementation of the protocol by bresser.

I thus modified hideki.c in the following ways, I don't know if it would be possible/wishable to integrate that further as a 'bresser' sensor on its own, and my solution may not be final at all (for instance there are still two redundant measurements for each transmission):

• see diff here

• At the beginning of hideki.c, the length of the incoming bits is evaluated to determine which sub-protocol to use. That step is already wrong with my bresser, because 89 bits instead of 90, so I force the sub protocol to be 'HIDEKI_TS04'

• Then there is the right shifting 2 bits

unsigned char bits1 = 0, bits2 = 0;
    int len = bitbuffer->bits_per_row[0];
    for(int i = 0; i < len; ++i) {
       bits2 = b[i] & 0x03;
       b[i] >>= 2;
       b[i] |= bits1 << 6;
       bits1 = bits2;
    }

• I de activated most of the tests but the parity bits, XOR check is still wrong, I guess this is due to the missing bit

[zuckschwerdt]

That sounds wrong. The protocol is [119] Bresser Weather Center 5-in-1, s.a. https://github.com/merbanan/rtl_433/blob/master/src/devices/bresser_5in1.c Do you have the most recent version of rtl_433 (i.e. is protocol 119 available)? If so maybe try https://triq.org/rtl_433/ANALYZE.html

[robertbiloute]

Maybe you're mixing up between '5in1' (station) and '5ch' (only a remote sensor with 5 channels) ? This protocol does not look like what I see in my packets at all, and Bresser explicitely told me it was using the HidekiTS04 protocol which is mentionned explicitely in hideki.c Also when I use the 119, I don't have any reception.

[zuckschwerdt]

Oh sorry. Yes, that makes sense now. Thanks for analyzing this new sensor!

Ideally we'd want to find a preamble, then sync on that to get the bits aligned in all cases, not just randomly add/remove bits ;) The need to pad bits is really not great. Since the XOR check starts at the second byte, we might instead be able to just ignore 6 bits (bitbuffer_extract_bytes()). I guess demanding a full 8/9 bit preamble wasn't flexible enough to start with ;)

Can you also look into a sample, maybe upload one as zip, is the preamble maybe weak or wrongly detected?

[robertbiloute]

I agree it's quite dirty, as said I can't even be sure my sensor is not faulty (an independant confirmation would be great I only have one of those)

Thx for your answer I'll look into that.

[robertbiloute]

Here are 4 samples in a zip (4 cu8 files with naming <something>_T<temperature>_H<hygrometry>_CH<channel nb>.cu8)

BRESSER_5ch_4samples.zip

I acquired them with rtl_433 -R 0 -c rtl_433_HIDEKITS04.conf -S all The rtl_433_HIDEKITS04.conf file is made with the decoder specs from hideki.c (long and short pulse duration etc..)

decoder {
name = HIDEKIT,
  modulation = OOK_DMC,
  short = 520,
  long = 1040,
  reset = 4000,
  tolerance=240,
  invert,
  get = {8}:SYNC+HEAD,
  get = @18:{8}:LEN,
  get = @27:{8}:Nr,
  get = @40:{4}:deg1,
  get = @36:{4}:deg.1,
  get = @45:{4}:deg10,
  get = @36:{4}:Hygro1%,
  get = @39:{4}:Hygro10%

}

(get features not important here, from a previous attempt)

There are three packets for each transmission, first one with 76 bits, 2 others redundant ones with 89 bits, the 3 packets have the same timestamp. The second and third packets fits with the description of TS04 protocol in hideki.c (once the 2 bits shifting is done and forgetting the missing bit)

[robertbiloute]

b.t.w IMO there is something odd about the exact location of data bits.

If you look at hideki.c preamble, you have:

Temp:
    11111001 0  11110101 0  01110011 1 01111010 1  11001100 0  01000011 1  01000110 1  00111111 0  00001001 0  00010111 0
    SYNC+HEAD P   RC cha P     LEN   P     Nr.? P   .1° 1°  P   10°  BV P   1%  10% P     ?     P     XOR   P     CRC   P
TS04:
    00000000  11111111  22222222  33333333  44444444  55555555  66666666  77777777  88888888 99999999
    SYNC+HEAD cha   RC     LEN        Nr.?    1° .1°  VB   10°   10%  1%     ?         XOR      CRC

Reading the bits "by hands", it is clear the "Temp" protocol is the one I have with the bresser sensor, not the TS04 one. Nevertheless, the "Temp" case in the following of the code does not read hygrometry, and forcing the "TS04" case on my bresser gives correct T and H (once the magic shift is applied)

[robertbiloute]

here in hideki.c, there is a test on the first octet, it has to be 0x9f ('0b10011111'). (I commented it in this modified version)

The first octet I read, translated from rtl_433 hexa value, is 11100100, but I add 2 0 bits to make it work, so I rather have 00111001.

If I understood things well, II should read 11111001 (time reverse 0x9f), and to mimick the thing perfectly I should shift 2bits but add 2 ones at the beginning.

TL;DR: it looks like the preamble is wrong, like I'm missing/the sensor is not emitting the first two 1s (which are the 2 last ones in incoming data)

[robertbiloute]

When adding two 1 bits instead of 2 0 bits after bit shift, the 0x9f test on the first octet is indeed passed, but the XOR and CRC errors remain

[spider0709]

Do you have any new information about it ? I'm having the same problem. I have 3 sensors and only one is recognized as an Hideki and works well with this definition.

If you need any information about this working sensor let me know.

[robertbiloute]

Nothing new, I have a usable version for my single sensor but I don't know what's happening there.

Your example suggests it's a random behaviour within this serie..

I don' know if it coud be useful but you can check wether my modified rtl_433 version succeeds reading your 2 other sensors (I modified the protocol 42). Now I solve my problem "adding" two bits, but who knows maybe some are lacking 1 bit, 3 bits..

[spider0709]

I've copied your hideki.c that you linked 2 posts ago and compiled rtl_433 new, but now none of my 3 sensors wil be recognized now.

Is is the correct source-code with your changes, or do you have another ?

[robertbiloute]

That's all I have, that really sounds like faulty sensors without even a reproducible problem

[spider0709]

I‘m not sure because I bought one of this sensors 3 days ago and it doesn‘t work as such as the second one I‘ve bought last year.

In my analysis I saw, that all three sensors sends the same bit-sequence, but only one is recognized.

[robertbiloute]

Could you try:

rtl_433 -R 0 -c rtl_433_HIDEKITS04.conf -S all

which will create a .cu8 file for each detected signal with the master rtl_433 (shouldn't matter though), and with this rtl_433_HIDEKITS04.conf:

decoder {
name = HIDEKIT,
  modulation = OOK_DMC,
  short = 520,
  long = 1040,
  reset = 4000,
  tolerance=240,
  invert
}

and post the cu8 files along with T/H reading ? Doing it on all 3 sensors would be great, we would have a sample of 4 sensors counting mine.

The conf file should allow the detection of pulses, preventing several checks that are not passed with protocol 42 (bit parity and such), so we have a raw signal.

[spider0709]

I will create this samples on each sensor including the workig sensor. On every sensor I will write T/H an channel down so that we are able to detect the informations.

[robertbiloute]

great, I'll try to look into it on my side.

[spider0709]

Attached you will find all three sensors with all 5 channels and for each channel 4 samples as cu8-file. Each file is named with chanel_temperature_humidity_samplex.cu8. I will hope this helps to decode the sensors. BresserSamples.zip

By the way, in your config the bit-sequences will be inverted ? Do I undestand it correctly ?

[robertbiloute]

So from what I see:

• The working sensor does work like expected, starting with a '11111001' octet as expected by hideki.c
• the 2 other non-working sensors both have the very same problem as mine has : they start with '111001' and just lack 2 1 bits to be inserted left.

Now that is what I conclude from reading the raw bits when I replay your cu8 files with rtl_433 and my custom conf file.

If that is right, using my modified hideki.c on the 2 non working sensors should work, as I saw the same problem on my sensor. But no, it doesn't, I just checked the stuff up to the beginning of hideki.c, and for some reasons it's reading 3 packets of only 1 bit each.. So it just stops on a length check.

I don't understand why things are so different using the modified hideki.c or my .conf file. Let me rephrase : Reading the cu8 file with rtl_433 and the custom conf file, everything is fine except for the bit shift, but reading it with the modified hideki.c (protocol 42) rtl_433 says it can't even receive more than one bit..

I'll have to dig some more on this one, but I don't see how I could have a hand on the bit buffer length, except by changing the decoder parameters (long pulse/short pulse), but then I know they are set to the right values so..

[spider0709]

What tool did you use for analyse this files ?

My first try to analyse the Signal was to do it with rtl_433 -R 0 -A.

This output gives a link where I‘m able to see the bit-sequence, but I‘m not able to find any information about channel, temperature or humidity.

[robertbiloute]

I'm using for instance:

rtl_433 -R 0 -c rtl_433_HIDEKITS04.conf -r Ch1_T19.9_H42_Sample1.cu8

to get hexadecimal data that I then turn into binary with python, for example:

In [3]: bin(int('e40973bce6507427faa5580 ',16))
Out[3]: '0b11100100000010010111001110111100111001100101000001110100001001111111101010100101010110000000'

with that setup, you may find your temperature and hygrometry, but first you have to add two 1 bits on the left (on the 2 faulty sensors).

For this T19.9 H42 sensor 2 file I get (adding 2 1 on the left):

11111001 0 00000100 1 01110011 1 01111001 1 10011001 0 10000011 1 01000010 0 11111111 0 10101001 0 10101100000 00
                                             9   9      1          2    4

(each 4 bits is read reversed)

and rtl_433 -R 42 -r Ch1_T19.9_H42_Sample1.cu8, with my modified version but it does not work.

[spider0709]

I‘m not shure, but in your config you invert the signal and in your code (from the Standard) the bits are also inverted. When I understand it correctly, then the bits are inverted twice. Should it converted once and not twice ?

[robertbiloute]

I'm either using the config file (just to trig on pulses and get data in hexa) OR the modified code (which is not working with your faulty sensors), so in both method the bits are inverted. I have created the config file to have a way of detecting the pulse with the same parameters as hideki.c but without all the error checks (preamble, parity bits, CRC, XOR, length...) In the case of your faulty sensors, even the length check is not passed, which I don't understand because with the config file I read the right length.

[spider0709]

Ok, I try to add a logging to see whta's happen in every step. I hope it will help

[zuckschwerdt]

Maybe it helps to write the cu8 to ook (rtl_433 -w Ch1_T19.9_H42_Sample1.ook Ch1_T19.9_H42_Sample1.cu8), you can then inspect the ook (drop on https://triq.org/pdv/) modify and read it again to see works.

It seems the sensor isn't stable on starting the send:

And that prevents detection of the first two bits (long mark, long space DMC):

I'd say we just shouldn't expect the first two bit to be there. Try to find even a short sync/header, but then start the extraction (parity, xor, crc) at the id/ch.

[spider0709]

Another question is if is possible to read the second or third sequence, because the second and third sequence begins with 11111001.

[zuckschwerdt]

I've implemented that idea (https://github.com/merbanan/rtl_433/blob/master/src/devices/hideki.c#L66-L79) and added a loop over all rows. Not perfect, but should be better now?

[spider0709]

It works for one of the non working sensors. In my samples it is Sensor1. The other one (Sensor2) seems not to work.

[zuckschwerdt]

I made a copy/paste mistake in the commit. Should be fixed now (git reset needed, I force pushed the fixed commit).

[spider0709]

It doesn't fix the not working sensor2. Now two of my sensors works fine.

[robertbiloute]

works for my sensor, thanks !

[zuckschwerdt]

@robertbiloute can you check if the changes 0ce3fcf make sense, and what's missing? It's based on your analysis and idea of the missing first bits:

• start decoding and parity check at bit 9 (skipping the sync), e.g. packet[3] is now packet[2]
• find the (uninverted) sync with full 9 bits (start = 9) or then 8, 7, 6 bits (start = 8, 7, 6)

[robertbiloute]

it does make sense and I don't see what's missing.

I just looked at some raw signals of all 3 sensors from spider0709 in parallel : I don't have much prior experience but boy they are so different.. It even looks like the carrier frequency is not the same on every sensors

(Top to bottom : working, not working sensor 1, not working sensor 2)

I find the sensor 1 signal is the weirdest, yet it's the sensor 2 that still does not work if I'm not mistakening. (does not look like a channel related difference)

[zuckschwerdt]

the weird look on sensor 1 is likely that the center frequency matches the carrier exactly, the sine you'd expect then looks like DC -- just a very slow slope. Not sure what's up with the sensor 2, what should be the carrier sine is just a mess. Could it be very nearly out of band? Or is it clipping?

[robertbiloute]

the weird look on sensor 1 is likely that the center frequency matches the carrier exactly, the sine you'd expect then looks like DC -- just a very slow slope.

makes sense indeed

Sensor 2 data actually look quite clean, just higher frequency, so maybe just a frequency bias ?

[zuckschwerdt]

Oh, ok, that looks good. I then would just look at the resulting .ook and see if the data is either mangled or just somehow different from what the decoder expects?

[robertbiloute]

Comparing sensor 1 and 2 ook I don't see a difference (great tool by the way!).

It was the same thing when I ran rtl_433 on the cu8 files with my custom .conf file (which has the same decoder parameters as hideki.c) : I do detect packets and they correspond to what I see on sensor 1 (they just lack 11 at the beginning), yet when I run rtl_433 with protocol 42 you modified, I detect nothing, and further logging tells me bitbuffer->bits_per_row[0] is only one (3 1-bit long packets..) I really don't get it, and if it affects this bitbuffer->bits_per_row[0] variable I don't see what could be modified at the hideki.c level.

[zuckschwerdt]

I'll take a look if something is amiss with the bittbuffer. Is there a file in the BresserSamples.zip that has this behaviour? Or can you upload one such file?

[robertbiloute]

(see plot below, top: sensor 1, bottom: sensor 2)

The raw data show the carrier of Sensor 2 has more offset than sensor 1

I don't understand this mHz scale in the plots, but from the bandwidth of 250 kHz you may estimate an offset of about 20 kHz, so 50 ppm or so, not so bad right ? I tried a detection on sensor 2 cu8 with some values of ppm correction, but no luck.

Also there is a kind of harmonic detection above the carrier.

[zuckschwerdt]

The mHz means that no metadata could be found. If you have the _250k in the filename (and maybe then _433.92M or such also) the display will be better.

[robertbiloute]

I'll take a look if something is amiss with the bittbuffer. Is there a file in the BresserSamples.zip that has this behaviour? Or can you upload one such file?

as far as I see every sensor2 files have a bitbuffer->bits_per_row of 1

[robertbiloute]

The mHz means that no metadata could be found. If you have the _250k in the filename (and maybe then _433.92M or such also) the display will be better.

ow OK thanks that explains it

[zuckschwerdt]

Ohh. It's just a single bad bit, the rest is fine in the second rwo (missing two sync bits)

[00] { 1} 80                                  : 1
[01] {89} 1b f6 8c 42 99 af 8b d8 05 56 43 80

…but I didn't remove the returns, so a single bad row ends the decoding. I'll fix that now. Thanks for patiently seeing this through!

[spider0709]

Should I test your changes on my not working sensor ?

[zuckschwerdt]

Should really all be fine now, with maybe a few missed signals now and then.

[robertbiloute]

splendid, also works on sensor 2 now

[robertbiloute]

I guess it's solved, thanks a lot @spider0709 it was just lacking some other sensors, and thanks a lot @zuckschwerdt that was really instructing

[spider0709]

Many thanks to @robertbiloute and @zuckschwerdt . All of my sensors are now working properly. What a great job at Christmas. What could it be a nicer gift ?

Have a good and healthy 2022.

[robertbiloute]

Ow right we were so focused we even forgot basic politeness : merry christmas and a better 2022 to you guys!