Custom firmware ATC

[Alex18081]

Hello, could you adapt your script for Custom firmware ATC? https://github.com/atc1441/ATC_MiThermometer Thanks!

[JsBergbau]

What exactly do you need?

[Alex18081]

Your script cannot receive data from the sensor with this firmware.

[shehrozeee]

As stated on the repo of the cust-firmware

The custom firmware sends every minute an update of advertising data on the UUID 0x181A with the Tempereature, Humidity and Battery data.

The format is like this:

Byte 5-10 mac in correct order

Byte 11-12 Temperature in int16

Byte 13 Humidity in percent

Byte 14 Battery in percent

Byte 15-16 Battery in mV uint16_t

Byte 17 frame packet counter

0x0e, 0x16, 0x1a, 0x18, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xaa, 0xaa, 0xbb, 0xcc, 0xdd, 0xdd, 0x00

You need to modify the LYWSD03MMC.py to work with this format.

for example

humidity=int.from_bytes(data[2:3],byteorder='little') will need to be changed to humidity=int.from_bytes(data[12:13],byteorder='little')

or something different based on how the data is received. Insert a breakpoint, watch the data array and figure out what is where and then read that I guess. Contribute back by making a pull request with conditional code or by making another public repo and share it with the author of the custom repo too so that he can link it on his repo and the next person does not have to put in the same effort

[JsBergbau]

It is not so easy to change. This custom firmware sends the measurements via Bluetooth advertisments. Therefore you have to be in scanmode all the time. I've discovered that with Raspberry PI Zero W (I think the other models will behave similiar) stops scanning mode after a few seconds when you maintain a connection to the sensor. So if you change to custom firmware, you should change all your sensors.

My script is based upon a connection to the device, so conditional code wouln't make so much sense I think. So if I use data from that sensors, I'll start a new Repo.

This custom firmware, which I really like even before installing it onto a single device, so thank you very much for sharing the link here, makes the device behaving like the older round Xiaomi Bluetooth sensor, image see here https://zsiti.eu/xiaomi-mijia-hygrothermo-v2-sensor-data-on-raspberry-pi/

In this link is also a reference to this repo https://github.com/ratcashdev/mitemp So correctly that project would be your base to start with.

However I've already thought about using that custom firmware for all my sensors. One advantage is that multiple Raspberry PIs could receive data from the same sensor which reduced reception problems because of too weak signals. You also could just move the sensors around like in a cellphone network, no need to find the right Raspberry PI and active the script there.

If you want that data send to influx you just could use the MAC-Adress as tag and a time resolution of 1s. Since influxdb "overwrites" the values with same timestamp you still would have only one value in your database. But since the mac adress as tag is not very clear which sensor is meant, an address translation would be nice. So you could send the data via MQTT. Node RED then receives it, translates MAC to a tag like "living room" and stores this data to influxdb. Via the trigger node with a timeout e.g. for 3s you can even prevent multiple requests to influxdb.

This sounds like a very nice project.

I have one concern. Since BLE uses three advertising channels (37, 38, 39, frequencies 2402 MHz, 2426 MHz and 2480 MHz) it could be that your bluetooth receiver is currently on the wrong channel when the advertisement is beeing trasmitted. So you miss that data. I'll give it a try and then report back.

[JsBergbau]

I've checked it. I've understood

The custom firmware sends every minute an update of advertising data on the UUID 0x181A with the Tempereature, Humidity and Battery data. the wrong way. Probably because original firmware sent every 5 minutes or so its current values encrypted via advertising packet.

However this custom firmware sends the data about once every second. Only the update of this data is done in the 10 seconds interval or every minute (customizable). So this makes it very interesting.

But beware that this is also a kind of privacy issue since the temperature and humidty can be read out by everyone. In connected mode no attacker can connect as long as you're connected.

[Alex18081]

I think in terms of privacy, temperature and humidity are the most harmless things that others can find out :) But in this mode, the processor load is low and more devices can be polled simultaneously. I tried to poll 5 sensors with a weak processor at different intervals. The processor was very busy, since the sensors do not always respond immediately, sometimes re-polling is required, over time one process was superimposed on another, which led to a complete system failure.

MfG

Alex :)

[JsBergbau]

It depends. If you configure them for a 10 seconds intervall you could to presence detection with them. E.g. if I go into my bathroom for brushing teeth I can see that in a small rise of the humidity and falling afterwards. With showering this effect can be seen much more. With BME280 you can even detect presence after a few seconds because of its high resolution.

But to your request, I think in one to 2 days I have the script fininished for supporting the ATC firmware. Thank you very much for the hint to this firmware.

[Alex18081]

Great news! I am grateful to you for your work.

[JsBergbau]

@Alex18081 Just a short info. Reading out ATC data works. It reads all sensors it is receiving. I'm currently implementing some useful stuff like configuring names for these sensors and the possibility to read out only these sensors. Of course there is still the calibration possibility. Since we get all sensors with one script instance this needs some other method than specifiying it on the command line.

Had to do a lot of other things the last days, so I hope I can get online the new version tomorrow. It is integrated in the "LYWSD03MMC.py" script and is fully backward compatible.

[Alex18081]

Excellent! Looking forward to trying it out :)

[JsBergbau]

Hi @Alex18081 the new version is finally out. Had some trouble because it stopped receiving data. But now problem is solved. Check out the Readme.md for all options and please report back your experiences. I've currently not the time to re-calibrate all my sensors, so I'll stay for some time on non ATC version. So I really appreciate your experiences and I'll switch all to ATC version since this saves a lot of battery.

[Alex18081]

Thank you very much. Unfortunately, I couldn't run the script in ATC mode. I get the same error on two different systems Traceback (most recent call last): File "./LYWSD03MMC.py", line 391, in import bluetooth._bluetooth as bluez ModuleNotFoundError: No module named 'bluetooth' in normal mode it works as before (not atc)

[JsBergbau]

Did you see the new section in the Readme? https://github.com/JsBergbau/MiTemperature2#requirements-for-reading-xiaomi-temperature-and-humidity-sensor-with-atc-firmware Please do these steps, then it should work.

[skibbipl]

Hi, I also followed the ATC section and installed all required modules yet still I get following error in ATC mode:

Traceback (most recent call last):
  File "./LYWSD03MMC.py.1", line 393, in <module>
    from bluetooth_utils import (toggle_device,
ModuleNotFoundError: No module named 'bluetooth_utils'

[JsBergbau]

As which user do you run it? When you run it as root, you have to install the requirements via sudo pip3 install pybluez If you run it as user via pip3 pybluez

[skibbipl]

root@rpi4b ~ # whoami
root
root@rpi4b ~ # pip3 install pybluez
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: pybluez in /usr/local/lib/python3.8/dist-packages (0.23)

[JsBergbau]

Ok, then try executing the script as root via ./LYWSD03MMC.py --atc

[skibbipl]

I'm running it as root

[JsBergbau]

Oh sorry, now I see the problem. It seems you have only copied the LYWSD03MMC.py Please also copy bluetooth_utils.py from the repository or download it via git clone https://github.com/JsBergbau/MiTemperature2

[skibbipl]

Yeah, my bad. Previously it was only one file so I just updated it. It works fine now. Thanks!

[eduperez]

Would it be possible to use the ATC firmware (and just wait for the announcements), but still handle the messages like the original firmware? This would allow to use the "Mi-Like" option from the ATC firmware.

[JsBergbau]

Theoretically this is possible. It depends on how complicated these Mi-Like frames are to decode. What's your usecase that you flash a custom firmware but then want to use the advertisments like in the original firmware?

If you find a good documentation about the Mi-Like firmware advertisments and post it here, I'll take a look.

[eduperez]

I want to use the ATC firmware, because of the configuration options and longer battery level; but I cannot use its format, because I need to integrate several sensors with several receivers. The source code from the ATC sensor where both types of messages are generated is at https://github.com/atc1441/ATC_MiThermometer/blob/master/ATC_Thermometer/ble.c

I think I will give this a try, and send a PR.

[JsBergbau]

Which receivers do you use? I mean it would be also an option if they'd support ATC format.

You don't have to create the full code. Just a written description of the Mi like Format would be great, because that saves some time compared to reading the specs out of the source code.

[eduperez]

This piece of code could decode both formats:

[...]
data_str = raw_packet_to_str(data)
atcIdentifier = data_str[6:10].upper()

if (atcIdentifier == "1A18"):
        ATCPaketMAC = data_str[10:22].upper()
        advNumber = data_str[-2:]
        temperature = int.from_bytes(bytearray.fromhex(data_str[22:26]),byteorder='big',signed=True) / 10.
        humidity = int(data_str[26:28], 16)
        batteryVoltage = int(data_str[30:34], 16) / 1000
        batteryPercent = int(data_str[28:30], 16)
elif (atcIdentifier == "95FE"):
        ATCPaketMAC = data_str[20:32].upper()
        advNumber = data_str[19:20]
        MiIdentifier = data_str[32:38].upper()
        if (MiIdentifier == "0D1004"):
                temperature    = int.from_bytes(bytearray.fromhex(data_str[38:42]),byteorder='little',signed=True) / 10.
                humidity       = int.from_bytes(bytearray.fromhex(data_str[42:46]),byteorder='little',signed=True) / 10.
                batteryVoltage = None
                batteryPercent = None
        elif (MiIdentifier == "0A1001"):
                temperature    = None
                humidity       = None
                batteryVoltage = None
                batteryPercent = int(data_str[38:40], 16)
        else:
                return
else:
        return

macStr = mac.replace(":","").upper()
[...]

Then, two packets must be merged into one update, as the MiLike option sends one packet with temperature and humidity, and another one with battery percent (battery voltage is never sent).

[JsBergbau]

Thanks for your code. I'll add it into the code. Before I have one question left, which save me a lot of time not having to try myself.

Have you tested with original firmware flashed sensors? As far as I know these sensors send these packets only every few minutes, so taking some time to test. These pakets are encrypted: So did you test what happens if your code receives one of those encrypted packets? Is there then gibberish decoded? If so how could it be prevented?

[eduperez]

I do not think my code is ready to be added "as is": it produces two different outputs, each of them with just one part of the info, and that might confuse the callback program. Besides, I haven't tested it with a device configured to output ATC's format.

I still have some sensors with the original firmware, I can try to see what messages they send.

[JsBergbau]

The two different Outputs are no problem. I just save the battery value in a variable and then when the packet with temperature arrives, I use that saved value to fill the battery percent value. So there is only one output left.

Would be great if you could test the behaviour of your code with original firmware devices.

[eduperez]

This is a sample of the raw packet sent by a device with the original firmware:

15020106111695fe30585b0501ea72d438c1a4280100

And this is what btmon --write hcitrace.snoop outputs:

> HCI Event: LE Meta Event (0x3e) plen 33                                                         #1 [hci0] 11.011106
      LE Advertising Report (0x02)
        Num reports: 1
        Event type: Connectable undirected - ADV_IND (0x00)
        Address type: Public (0x00)
        Address: A4:C1:38:D4:72:EA (Telink Semiconductor (Taipei) Co. Ltd.)
        Data length: 21
        Flags: 0x06
          LE General Discoverable Mode
          BR/EDR Not Supported
        Service Data (UUID 0xfe95): 30585b0501ea72d438c1a4280100
        RSSI: -86 dBm (0xaa)

That corresponds to a temperature of 23.2C and a humidity of 48%.

[JsBergbau]

I'm quite confused now. According to https://github.com/esphome/feature-requests/issues/552 the data is encrypted on a sensor with original firmware, so did you check that 23.2 °C and 48 % humidity is the value actually shown on your sensor?

[eduperez]

Yes, I switched on an out-of-the-box sensor, read the temperature and humidity (manually) from the display, and then copied the output from btmon. Values could be slightly off, however: the sensor is in another room, and could have changed while I moved from one room to another.

I am trying to reproduce this, but btmon does not output any packet now, from any device at all... I must have broken something.

[JsBergbau]

Try running hcitool lescan --duplicate in background then btmon should output packets again. But if any other process connects to a bluetooth device, then you have to restart hcitool lescan --duplicate Or you can use my script with --atc --watchdogtimer 5 this enables LEScanning automatically if it stops by any reason.

[eduperez]

Perhaps I do not understand how this is supposed to work, because nothing makes sense any more...

I repeated the same test again, and the sensor was sending exactly the same message over and over again, even if the temperature and humidity changed. I re-paired the sensor with the phone again (that is supposed to change the encryption key) and now the sensor sends a different message, but it is the same message always.

[JsBergbau]

As far as I know it sends the encrypted temperature and humidity only every X minutes. Perhaps you didn't wait long enough? This Mi advertisments are quite complicated, thats why I didn't focus on them.

[dbluxo]

@JsBergbau Will it also be possible to get the battery charge percentage when using the custom firmware ATC?

[JsBergbau]

@dbluxo Of course it is. Just use -b battery switch and battery in percentage is printed with ATC firmware.

[dbluxo]

@JsBergbau awesome, thanks! 🎆