Mi-like advertising data structure.

[vevsvevs]

Won't it break your concept too much, if ADV-data will be structured as for an MI-like device? I mean using 0x95, 0xFE UUID, some device identification and predefined data points?

[atc1441]

It can be edit to it without problems, but the 0x95FE is a xiaomi branded uuid so i tried to avoid it.

Using that one on the other hand would make ESPhome directly vompatible so i understand your point

[TheNitek]

I would also love this. Since Xiaomi used basically the same format for all their sensor it would be very convenient to just stick with one decoding routine that works for all, including the "freed" ones with custom firmware.

[atc1441]

If there is no copyright problem im willing to implement it

[TheNitek]

Well I am no lawyer, but I don't see how the structure of the advertisment would be copyrighted. I don't think it has the necessary "Schöpfunghöhe" 😃

[atc1441]

Ok sounds good. The problem i have is with the 0x95FE as that is "bought" by xiaomi.

[vevsvevs]

@atc1441 well, it's still the Xiaomi device, right? ;) I'm not a lawyer either, but I believe that until you don't use this UID for custom commercial hardware - there is no possible legal issue.

[atc1441]

Added the beginning of advertising like an Mi Thermometer, https://github.com/atc1441/ATC_MiThermometer/blob/master/ATC_Thermometer/ble.c#L167

but i didn't found a good example of how the Mi protocol looks like, does someone has an example?

That way it works already, i think i will make it a setting later so you can change on runtime what Service uuid should do the advertising.

[vevsvevs]

@atc1441, gathering information in ESPHome repo gave me this:

ADV-packet structure:
    Byte 0:     data/capability/encryption flag
    Byte 1:     ?
    Byte 2..3:  device-id
    Byte 4:     frame count
    Byte 5-10:  reversed MAC
    Byte 11...  data points

Data point structure:
    Byte 0:     type
    Byte 1:     fixed 0x10
    Byte 2:     length
    Byte 3..3+len-1: data point value

Data type IDs and lengths:
    0x04:       temperature 0.1°C (int16_t)
    0x06:       humidity 0.1% (int16_t)
    0x0A:       battery 1% (uint8_t)

I only mentioned the needed data type IDs. The battery voltage wasn't seen in the wild yet, but last known data type ID is 0x17, so something like 0x1a should be fine, I think :) Data/capability/encryption flag should be 0x50 in our case.

[ahpohl]

I now implemented the custom advertising structure here. It was very easy to implement compared to the original Mi stuff. I am not sure the "Mi like" advertising structure will work out of the box with the xiaomi_lywsd03mmc component. I have to think it through how the service data needs to look like.

[vevsvevs]

@ahpohl thank you very much! I've tried to implement my own integration pretty much the same way, but I'm stuck with compilation error due to lack of C-syntax understanding :)

@atc1441 if you didn't bother yet too much with my request, maybe it should be canceled now, this implementation is more than enough to use. Sorry for disturbing you.

[ahpohl]

@vevsvevs Advantage to support the "Mi Like" advertising would be no further PRs to ESPHome would be necessary. I think your request has already been implemented by @atc1441 . We can easily change the custom firmware to make it fit (thanks to open source!)

[atc1441]

Great to see its implemented now,

Was just done implementing the Mi Like advertising....

You can set it from the Flasher simply with a button click.

I am not shure if the format is correct of the mi advertising escpecially the length.

https://github.com/atc1441/ATC_MiThermometer/blob/master/ATC_Thermometer/ble.c#L24

Would love if someone with ESPhome can test it.

It does not Advertise Battery mV as its to long then.

Also as adition the LCD will show Humidity and battery percent in change now indicating via the Battery symbol.

[ahpohl]

@atc1441 I can test the "Mi Like" function and I tell you in a minute how the advertising needs to look like in order to make it work out of the box with ESPHome

[atc1441]

Thank you, a real life example data packet would be nice to have including the real values to know how it exactly should look like

[ahpohl]

Another topic: I pressed the 10 minute interval button a couple of times and currently it is receiving data approx every 2 min. Have you tested the 10 min. interval?

[atc1441]

didnt tested that no, is the counter increasing on every advertising ?

it could be that the advertisind_delay simply has an overflow so it can not count so long, will change it if so

[atc1441]

Ok thank you will change the delay values from us to seconds so they dont overflow. Thought about that a while ago but never encountered a problem till now ^^

[ahpohl]

The ATC MiTemperature component has a duplicate packet filter and rejects duplicate packets. I see quite a lot of them with the same frame counter, but then after approx. 2 min the frame counter increases and the next packet is processed. So yes, maybe its a variable overflow issue in the firmware which needs fixing.

[atc1441]

Maybe changing the delay variable from uint32 to long does help as well?! Should be a long enough delay then just not shure if TC32 compiler does support it will test.

[ahpohl]

I switched to the "Mi Like" advertising. This is what I get:

[15:43:37][VV][esp32_ble_tracker:343]:   Address: A4:C1:38:4E:16:78 (PUBLIC)
[15:43:37][VV][esp32_ble_tracker:345]:   RSSI: -55
[15:43:37][VV][esp32_ble_tracker:346]:   Name: 'ATC_4E1678'
[15:43:37][VV][esp32_ble_tracker:363]:   Service data:
[15:43:37][VV][esp32_ble_tracker:364]:     UUID: 95:FE
[15:43:37][VV][esp32_ble_tracker:365]:     Data: 50.00.AB.BE.04.78.16.4E.38.C1.A4.04.10.02.00.EE.06.10.02.02.44.0A.10.01.52 (25)
[15:43:37][VV][esp32_ble_tracker:368]: Adv data: 1C.16.FE.95.50.00.AB.BE.04.78.16.4E.38.C1.A4.04.10.02.00.EE.06.10.02.02.44.0A.10.01.52.0B.09.41.54.43.5F.34.45.31.36.37.38 (41)
[15:43:37][VV][xiaomi_lywsd03mmc:024]: parse_device(): MAC address A4:C1:38:4E:16:78 found.
[15:43:37][VV][xiaomi_ble:108]: parse_xiaomi_header(): no service data UUID magic bytes.

For comparison an original packet:

[12:03:39][VV][esp32_ble_tracker:321]:     UUID: FE:FE95
[12:03:39][VV][esp32_ble_tracker:322]:     Data: 58.58.5B.05.17.B7.34.8C.38.C1.A4.EE.37.C3.AC.10.46.00.00.FC.27.F0.A5 (23)
[12:03:39][VV][esp32_ble_tracker:325]: Adv data: 02.01.06.1A.16.95.FE.58.58.5B.05.17.B7.34.8C.38.C1.A4.EE.37.C3.AC.10.46.00.00.FC.27.F0.A5.0B.09.4C.59.57.53.44.30.33.4D.4D.43 (42)
[12:03:40][VV][xiaomi_ble:118]: parse_xiaomi_service_data(): encrypted packet received.

It seems that the magic bytes have the wrong endianess. The same was actually true for the custom firmware advertising. I had to swap UUID 0x181A into UUID 0x1A18 in order to be recognized.

[atc1441]

Changed the integers now to long long and that should fix it, can you please try it out

[atc1441]

Looks like your stock data/ original packet is from an LYWSD03MMC but we need a original packet from another Xiaomi thermometer without encryption.

[ahpohl]

The advertising packets have been explained here: https://github.com/Magalex2x14/LYWSD03MMC-info.

Looks like you stock data/ original packet is from an LYWSD03MMC but we need a original packet from another Xiaomi thermometer without encytion.

Yes, I'll did through some old logs.

[vevsvevs]

@atc1441 here is a data from CGG1:

5030470386FD4910342D58061002B701

[atc1441]

@vevsvevs thank thats perfect.

Changed the endianes of the service, that was indeed of

[ahpohl]

The CGG1 sends the following unencrypted advertisement:

50.20. 47.03. 16. 67.3A.10.34.2D.58. 0D.10.04.E1.00.87.02. 0A.10.01.15 (22) 
50.30. 47.03. D0. 7A.4C.10.34.2D.58. 04.10.02.E9.00.       0A.10.01.10 (20)
ctrl   type   cnt        mac rev     data 1                data 2

For the LYWSD03MMC the type would be 5B 05. The frame control bytes indicate the packet is not encrypted, which I need because I need to skip decryption.

The flag 0x50 at byte 1 get checked like this:

  result.has_data = (raw[0] & 0x40) ? true : false;
  result.has_capability = (raw[0] & 0x20) ? true : false;
  result.has_encryption = (raw[0] & 0x08) ? true : false;

[ahpohl]

The temperature, humidity and battery data format is like this (data 1 and data 2 in last comment):

  // temperature, 2 bytes, 16-bit signed integer (LE), 0.1 °C
  else if ((raw[0] == 0x04) && (data_length == 2)) {
    const int16_t temperature = uint16_t(data[0]) | (uint16_t(data[1]) << 8);
    result.temperature = temperature / 10.0f;
  }
  // humidity, 2 bytes, 16-bit signed integer (LE), 0.1 %
  else if ((raw[0] == 0x06) && (data_length == 2)) {
    const int16_t humidity = uint16_t(data[0]) | (uint16_t(data[1]) << 8);
    result.humidity = humidity / 10.0f;
  }
  // battery, 1 byte, 8-bit unsigned integer, 1 %
  else if ((raw[0] == 0x0A) && (data_length == 1)) {
    result.battery_level = data[0];
  }
  // temperature + humidity, 4 bytes, 16-bit signed integer (LE) each, 0.1 °C, 0.1 %
  else if ((raw[0] == 0x0D) && (data_length == 4)) {
    const int16_t temperature = uint16_t(data[0]) | (uint16_t(data[1]) << 8);
    const int16_t humidity = uint16_t(data[2]) | (uint16_t(data[3]) << 8);
    result.temperature = temperature / 10.0f;
    result.humidity = humidity / 10.0f;
  }

[atc1441]

The custom Mi-like version should be compatible now

[atc1441]

Ah could be that temp and humi is still endianes switched ^^ please someone test it

[atc1441]

but will also change it so it use the combined 0x0D type then the data is shorter.

Changed the endianes now and updated it, so with luck its now like it should be :)

[vevsvevs]

For the LYWSD03MMC the type would be 5B 05

Maybe type (device id) shouldn't be the same as the original firmware, and better to alter parsing in your integration?

[atc1441]

Yeah type id is now AB BE just to differ it from others

[ahpohl]

Maybe type (device id) shouldn't be the same as the original firmware, and better to alter parsing in your integration?

With the "Mi like" advertising I think it should behave exactly like a Xiaomi device and work out of the box with the current integration. The custom advertisement is supported by the new component "ATC MiThermometer" with all references to Xiaomi removed. It doesn't even need xiaomi_ble anymore.

Here is what I get with the firmware uploaded 15 min ago. It still doesn't get the magic bytes because they are at the wrong position:

 [16:25:28][VV][esp32_ble_tracker:326]: Parse Result:
[16:25:28][VV][esp32_ble_tracker:343]:   Address: A4:C1:38:4E:16:78 (PUBLIC)
[16:25:28][VV][esp32_ble_tracker:345]:   RSSI: -58
[16:25:28][VV][esp32_ble_tracker:346]:   Name: 'ATC_4E1678'
[16:25:28][VV][esp32_ble_tracker:363]:   Service data:
[16:25:28][VV][esp32_ble_tracker:364]:     UUID: FE:95
[16:25:28][VV][esp32_ble_tracker:365]:     Data: 50.00.AB.BE.03.78.16.4E.38.C1.A4.04.10.02.00.F0.06.10.02.02.44.0A.10.01.52 (25)
[16:25:28][VV][esp32_ble_tracker:368]: Adv data: 1C.16.95.FE.50.00.AB.BE.03.78.16.4E.38.C1.A4.04.10.02.00.F0.06.10.02.02.44.0A.10.01.52.0B.09.41.54.43.5F.34.45.31.36.37.38 (41)
[16:25:28][VV][xiaomi_lywsd03mmc:024]: parse_device(): MAC address A4:C1:38:4E:16:78 found.
[16:25:28][VV][xiaomi_ble:172]: parse_xiaomi_header(): unknown device, no magic bytes.

[TheNitek]

Maybe type (device id) shouldn't be the same as the original firmware, and better to alter parsing in your integration?

I think it should stay the same. It's still the same just the encryption bit is not set and not encrypted.

[vevsvevs]

because they are at the wrong position:

I believe, not because of position, but it's AB:BE instead of 5B:05 ;)

@atc1441 maybe as you implemented selectable advertising_type, indeed there is no reason alter the device id for mi-like mode?

[atc1441]

if the parser can handle the same id without encryption i will set it to the stock id

[atc1441]

Its now at the stock id updated in the repo

[ahpohl]

yes, I think it does. The parser depends on the three flags data, capability and encryption being evaluated correctly. A value of 0x50 in the first frame control byte (the second is not checked) gives us true, false, false which will work. I am trying to remember what beginning of the service data means up to the point where the data packet starts

[TheNitek]

It works with my custom smarthome now without modification! Perfect!

[ahpohl]

The original adv packet from above:

02.01.06          1A         16        95.FE          58.58        5B.05         17          B7.34.8C.38.C1.A4        EE.37.C3.AC.10.46.00.00.FC.27.F0.A5.0B.09.4C.59.57.53.44.30.33.4D.4D.43 (42)
BR/EDR+LE flags   len        AD-type   Xiaomi UUID    Frame ctrl   device type   Frame cnt   mac reversed             start of encrypted payload

So great it already seems to work!

[DoeWayne]

Does that mean that it will work again with the Xiaomi Home application ? Absolut great work Aaron, many thanks to all of you

[atc1441]

It should work with any Xiaomi like implementation on other Systems but not the Mi Home App itself

Now only the 10 minute Advertising intervall needs to be tested.

Right now the settings will be back to default on battery removing, i am thinking about adding a storage in flash but without it its kinda failsave

[vevsvevs]

@DoeWayne it won't, of course.

[vevsvevs]

Yes should work with any Xiaomi like implementation on other Systems.

He asked about the original app, not other integrations to the alternative smarthome systems :)

[DoeWayne]

@vevsvevs I just recognized it, what a shame, so need to find another application on the mobile which should receive all the data. I'm just on the way to order another 20 devices.

[vevsvevs]

@atc1441 btw, there is some kind of problem with the battery icon state, tracing now..

[ahpohl]

Does the frame counter increase properly? I get lots of duplicate packets but not the data. My frame counter is at 0x0B (11, byte 5) for quite some time now (10 s interval setting)

[17:14:25][VV][esp32_ble_tracker:326]: Parse Result:
[17:14:25][VV][esp32_ble_tracker:343]:   Address: A4:C1:38:4E:16:78 (PUBLIC)
[17:14:25][VV][esp32_ble_tracker:345]:   RSSI: -53
[17:14:25][VV][esp32_ble_tracker:346]:   Name: 'ATC_4E1678'
[17:14:25][VV][esp32_ble_tracker:363]:   Service data:
[17:14:25][VV][esp32_ble_tracker:364]:     UUID: FE:95
[17:14:25][VV][esp32_ble_tracker:365]:     Data: 50.30.5B.05.0B.78.16.4E.38.C1.A4.0D.10.04.F0.00.3A.02.0A.10.01.52 (22)
[17:14:25][VV][esp32_ble_tracker:368]: Adv data: 19.16.95.FE.50.30.5B.05.0B.78.16.4E.38.C1.A4.0D.10.04.F0.00.3A.02.0A.10.01.52.0B.09.41.54.43.5F.34.45.31.36.37.38 (38)
[17:14:25][VV][xiaomi_lywsd03mmc:024]: parse_device(): MAC address A4:C1:38:4E:16:78 found.
[17:14:25][VV][xiaomi_ble:124]: parse_xiaomi_header(): duplicate data packet received (11).

[DoeWayne]

Yes it does not change back to Hummidy level

[atc1441]

@vevsvevs the battery i con will be on when the battery status is shown in the humidity value section.

Thats so you know how full the battery is

[atc1441]

Ok. Yeah something is off there.

Will check