Support LYWSD03MMC new mijia temp/humidity sensor

[aidbish]

Describe the problem you have/What new integration you would like Add support for the mijia Temperature/Humidity Sensor LYWSD03MMC

Please describe your use case for this integration and alternatives you've tried: These are an updated version of the sensors from xiaomi mijia, and are small compact devices that can be used in discreet places

Additional context Work has been done in a number of places to support them, that maybe of assistance https://github.com/custom-components/sensor.mitemp_bt/issues/7

https://github.com/JsBergbau/MiTemperature2/

thankyou

[Trillon008]

The LYWSD03MMC is incredible value for the price and could fit gloriously in any smart home. But, following the links you provided it seems it doesn't work like any other Xioami device and doesn't broadcast his data but instead requires a direct connection. This will probably cause issues when multiples devices are concerned and ultimately consume more battery power.

[vevsvevs]

@Trillon008 no, it sends ADV-packets as well, but they're encrypted :)

[Joshfindit]

(Just got mine a couple hours ago, and super interested in pulling the data in)

There is a fundamental shift in how it operates (seems to require pairing where the previous versions broadcast to the open air), but the linked python script is able to read the data without issue. It seems likely to me that a new xiaomi_lywsd03mmc sensor would have to be created that supports pairing (and hopefully supports simultaneous pairing of, say, 3-6 devices) and then listens for data from the paired devices.

[Joshfindit]

Likely standard BLE encryption, and since esphome is the master device, there would be no need to break the encryption

[vevsvevs]

But esp32_ble_tracker has no master/slave approach, it passively listening.

[Joshfindit]

@vevsvevs Yes, that’s what I meant: there would have to be a separate esp32_ble_tracker that supports pairing in order to provide functionality for this sensor

[Joshfindit]

Apologies if this looks like advertising, I’m really just trying to help dev results. They went on sale on banggood https://banggood.app.link/3oEGAJptH3

[Monacoslo]

Anything new regarding the use in HA?

[vevsvevs]

Meanwhile, the ADV-payload decryption method for this sensor is already known. I hope some high-skilled contributor will be so kind to implement support of this sensor soon :)

[aidbish]

@vevsvevs Yes it was found, but only a few hours ago. Give people a chance to look into it

[gbarneda]

Anything new? Meanwhile i've found this but I don't know how to use it

[uKL]

Anything new? Meanwhile i've found this but I don't know how to use it

This one is based on GATT connection and notifications while ESPHome way is to listen to advertised packets.

[rufik]

I think HA custom component _mitempbt has found some way to fetch data passively, see https://github.com/custom-components/sensor.mitemp_bt/issues/7#issuecomment-597476784 It looks like some paring with MiHome app is required first, then encrypt key extraction, but done once can be used in esphome config I think :)

[ahpohl]

I bought 8 of the new LYWSD03MMC mijia temp/humidity sensors and 3 ESP32 dev boards to be used with ESPHome. I would really like to get the sensors directly supported in ESPHome. I have collected a full Android HCI dump while pairing the devices in the Xiaomi Home App and looked at it in Wireshark , but don't know how to proceed from here. I would really appreciate if somebody more skilled than me could step in.

It looks like some paring with MiHome app is required first, then encrypt key extraction, but done once can be used in esphome config I think

It would be really nice if that would be possible and hopefully the encryption key does not change over time.

[Joshfindit]

It looks like some paring with MiHome app is required first, then encrypt key extraction, but done once can be used in esphome config I think

It would be really nice if that would be possible and hopefully the encryption key does not change over time.

Hopefully with the right people involved, ESPHome could be the one to do the initial pairing, making extracting the key much simpler.

[ahpohl]

What are the next steps? I can read the temperature and humidity with this python script, but only when connecting to the device. I collected a clean dump of the conversation and can see the raw sensor values in the Notification, Handle 0x0036 as described in the README.

But that is not what I need. I need to passively collect the data simultaneously from eight sensors with just two ESP32 (flashed with ESPHome connected via MQTT to HA and Grafana). As I understand, the device is constantly sending the data, but the data is encrypted. How do I extract the encryption key from the raw dump?

Meanwhile, the ADV-payload decryption method for this sensor is already known. I hope some high-skilled contributor will be so kind to implement support of this sensor soon :)

It sounds as it is actually possible!

[ahpohl]

Obtaining the bind-key for ADV-payload decryption is quite simple following the guides described in the above mentioned project custom-components/sensor.mitemp_bt. I have extracted the bind-keys for all my eight LYWSD03MMC sensors and they are working fine in HA. However, I want to distribute them around the house and therefore need the ESP32s with ESPHome for the bluetooth connectivity.

I had a quick look at the python code in sensor.py, which emulates hcitool, constantly scans for the broadcasted ADV packets and automatically decrypts the payload with the temperature and humidity values. This is basically what we need without the HA code, so an experienced programmer needs to translate the existing python code into Arduino C++ code to be included into ESPHome. Unfortunately I am just starting with bluetooth programming, so somebody more experienced is welcomed to step in.

I hope the sensor will get eventually supported in ESPHome.

[clau-bucur]

I also thought I'd need ESP32 or some more bluetooth adapters to be able to cover the entire house. But it seems I don't, after all. One USB dongle works perfectly. The farthest sensor from the dongle is about 5m apart on a different level (about 1.5m height difference), separated by concrete and brick walls. I was really surprised it was read. However your situation might be entirely different, but you can give it a try. You can also use multiple Bluetooth dongles with mitemp_bt so you can have USB-to-Ethernet extenders (I use these, they work nicely) and place some around the house.

[ahpohl]

you can have USB-to-Ethernet extenders (I use these, they work nicely) and place some around the house.

Interesting, I didn't know that it is possible to extend the USB via Ethernet cables. It would indeed be a possibility for me as I have unused RJ45 outlets in every room. But I prefer to use the ESP32s as they are more versatile and also more fun (and I already have bought all hardware).

I started to look into the ESPHome pyhton and C++ code and how to implement the new sensor component. It seems that the esp32_ble_tracker component needs to modified to include the payload decryption and also a new sensor configuration needs to be created.

I am just starting to learn ESPHome and bluetooth programming. So it will take me a while to figure it out if I have to do it all by myself. But I'll give it a try.

[Magalex2x14]

@ahpohl Feel free to ask any questions about the operation of the custom component mentioned above and the technical details about LYWSD03MMC - I will help you all I can.

[ahpohl]

@Magalex2x14 Thanks for offering your help. In the meantime I have successfully created a standalone python script mitemp_standalone.py from your HA custom component sensor.py to get familiar with bluetooth low energy programming.

The esp32_ble_tracker component is another story though. As far as I can tell, the function ESPBTDevice::parse_adv_ does the parsing of the packets. It needs a lot more code to detect the encrypted payload, decrypt and parse the output, and probably more classes or functions. I found an AES library for the ESP32, not sure if it is compatible with FreeRTOS and will be of any use.

Which IDE do you recommend for coding? How do you test the new code? I find it difficult to develop for ESP32 when I cannot use the Arduino IDE directly.

[ahpohl]

The Crypto++ library includes AES-CCM with CBC-MAC and is more likely what we need. Another option would be Botan, which looks more modern and easier to start with.

[Magalex2x14]

@ahpohl I use Visual Studio Code with addons like python linters, formatters, etc.

For VSCode there are also C++ and Platformio addons, and I used them when I made a custom component for working with oled display on my ESP32 board. Unfortunately, this is where my knowledge of ESPHome architecture ends... At the moment, I'm a bad assistant in this...

Just in case, here is a link to the esphome development environment setting up.

[Monacoslo]

https://community.home-assistant.io/t/xiaomi-passive-ble-monitor-sensor-platform/177352/144

this is great component, but I need working solution for ESP32...

[ahpohl]

Development of a xiaomi_lywsd03mmc custom component for ESPHome has been initiated. The C++ decryptor works and the code can be used to extend xiaomi_ble. I already created the new sensor type xiaomi_lywsd03mmc, but unfortunately the new platform is not recognised during build.

[Monacoslo]

even in 1.5 dev not?

[ahpohl]

even in 1.5 dev not?

@Monacoslo: Could you please explain what you mean with this comment?

I created a new platform xiaomi_lywsd03mmc (in a new brach with the same name) but get this when compiling the firmware

INFO Reading configuration livingroom.yaml...
Failed config

sensor.xiaomi_lywsd03mmc: [source livingroom.yaml:26]

  Platform not found: 'sensor.xiaomi_lywsd03mmc'.
  platform: xiaomi_lywsd03mmc
  mac_address: A4:C1:38:8C:34:B7
  temperature: 
    name: Temperature
  humidity: 
    name: Humidity
  battery_level: 
    name: Battery Level

EDIT: problem solved. I was not installing the new xiaomi_lywsd03mmc platform into my virtualenv after creating it. Now it recognizes the new platform!

[ppanagiotis]

even in 1.5 dev not?

@Monacoslo: Could you please explain what you mean with this comment?

I created a new platform xiaomi_lywsd03mmc (in a new brach with the same name) but get this when compiling the firmware

INFO Reading configuration livingroom.yaml...
Failed config

sensor.xiaomi_lywsd03mmc: [source livingroom.yaml:26]

  Platform not found: 'sensor.xiaomi_lywsd03mmc'.
  platform: xiaomi_lywsd03mmc
  mac_address: A4:C1:38:8C:34:B7
  temperature: 
    name: Temperature
  humidity: 
    name: Humidity
  battery_level: 
    name: Battery Level

EDIT: problem solved. I was not installing the new xiaomi_lywsd03mmc platform into my virtualenv after creating it. Now it recognizes the new platform!

I compile the firmware using your branch but the response is empty: xiaomi_lywsd03mmc:013]: Temperature 'Temperature' Are there any other steps or settings I have to to/add? Thank you very much!

[ahpohl]

@ppanagiotis its not quite finished yet. The new platform is currently just a template. A lot more work needs to be done, e.g. creating a new config item encryption_key, adding the decryptor code etc.

[Monacoslo]

I am using EspHome dev component for HA and esphome 1.5 dev firmware

and for example for dev components for mitsubishi I had to use this one instead of official version, so I am asking if you tried also that option.

[ahpohl]

I am using EspHome dev component for HA and esphome 1.5 dev firmware

@Monacoslo Thanks for clarifying. Support for Xiaomi LYWSD03MMC is not implemented yet, also not in the dev branch. But I am working on it to make it happen.

[ahpohl]

Hi, this is a short progress report.

The "offline" C++ decryptor has been implemented using Crypto++ as the AES engine. I realize that this library is not portable to PlatformIO. Alternatively, there is the Arduino Cryptography Library directly supported by PlatformIO and hence can be easily included in ESPHome. Unfortunately, the AES-CCM mode (Counter with CBC-MAC) is currently not supported by the library. I have opened a support ticket #52 and hoping for an answer.

Anybody knows an alternative Crypto library for PlatformIO with support for AES-CCM?

[Magalex2x14]

@ahpohl maybe there is something useful on this link? I myself have not watched in detail yet.

[ahpohl]

@Magalex2x14 I went through the library search already. The problem is that most small libraries do not implement the authentication (AEAD). The only library I found that does is Crypto for Arduino. However, Crypto doesn't implement the CBC-MAC mode. Unfortunately my cryptographic skills are not sufficient to extend the library myself, although it should be trivial to implement for an experienced cryptographer. The other popular authentication algorithms ChaChaPoly, EAX and GCM are already supported. It seems that AES-CCM mode, which BLE uses, is kind of depreciated.

[endrec]

Espressif has a library which supports CCM on the ESP32, but I have no idea how portable it is: https://github.com/espressif/arduino-esp32/blob/master/tools/sdk/include/mbedtls/mbedtls/cipher.h

[ahpohl]

@endrec Thanks for providing this link. It looks promising.

If mbedtls in the Expressif SDK is the same as mbedtls in PlatformIO, then it can be easily included: In sensor.py I added: cg.add_library('mbedtls', 'cdf462088d')

Looking for mbedtls library in registry
LibraryManager: Installing id=2509 @ cdf462088d
mbedtls @ cdf462088d has been successfully installed!

Looks good. Next step: find out how to use it!

[ahpohl]

I created a mbedtls.c test according to the library documentation. As mbedtls is part of the Arduino ESP32 core, the sketch runs also directly on the NodeMCU without the need to install an extra library.

gcc -g -o mbedtls mbedtls.c -lmbedcrypto

Name       : AES-128 CCM BLE ADV
Key        : E9EFAA6873F9F9C87A5E75A5F814801C
Iv         : 78164E38C1A45B053D2E0000
Cipher     : DA616677D5
Plaintext  : 041002D300
Tag        : 92982352
Authenticated decryption failed.
Plaintext  : 0000000000

Unfortunately, the code doesn't work yet. I also tried other test vectors from Crypto++ and the NIST CCM-MAC specification (in the source comments). Something must be wrong in how I use the library. I couldn't find good examples though.

Anyboy knows mbedtls well to review my code?

EDIT: the decryption fails because mbedtls expects a different tag. The correct tag for this test vector would be 9F1F0F10 instead of 92982352. What is the relationship between those tags and how to convert between the two? The tag original tag is given in the BLE ADV payload and is correct for the Crypto++ library. What is mbedtls doing differently? I opened a support ticket on the Arm MBED forum

[coldpenguin]

I thought there was nothing happening on this, turns out the thread wasn't refreshing correctly :( So good news and bad news from my point, I was at the point of creating a fork ready to create a draft pull request against esphome :). I hadn't noticed ahohl's comments.

+'ves 1) I have re-factored the xiaomi_ble code slightly so that the headers are parsed away from the message. This means that there is less code in the 'loop' I was getting some timeout warnings when I had three sensors (however, I think it might have been the ssd1306 instead) 2) I have the code decoding the temperatures, humidity and battery level. :)

-'ves 1) The esp32 devices seem to be missing some packets.:( . Even without my mods, with just logging on the xiaomi_ble I don't seem to be getting all the data packets. Maybe the esp32 just doesn't have as good an aerial? Maybe this isn't a problem for me, I was intending on running multiple esp's with a total of 9 xiaomi devices, so someone might get the data.

2) I need to check further , but I suspect that the problem @ahpohl is having is the same as I had, the authentication part of the decode just isn't working. I have gotten around this by using internal funcitons.

3) The dev branch of esphome is a lot different to the master branch! There are no xiaomi devices whatsoever! Harrumph.

[ahpohl]

@coldpenguin great news I am not the only one fighting with this!

Currently I am struggling with mbedtls. I have fully working authentication with Crypto++ as the AES library. The authenticated data is just one byte "\x11" and it seems to work (credit to @Magalex2x14 who did all the hard decoding work).

Any chance to embed Crypto++ into ESPHome? Unfortunately Arduino Crypto does not support AES-CCM. Crypto 0.2.0 is already a dependency in ESPHome and the library would be easy to use. @coldpenguin Which library are you using? Is your code already in your fork? My code lives here: https://github.com/ahpohl/esphome/tree/xiaomi_lywsd03mmc

[coldpenguin]

Taken from influxdb within home-assistant :)

This is with two ESP32s listening to three 003mmc's, at different ends of the house. About 10am this morning I disabled the verbose logging, and the number of received packets has increased to a usable amount (depending on use-case, but every so often one is missed). Ideally, I would like to decrease the advertisement time, but that time must be in the firmware or an obscured setting somewhere.

My code is here: https://github.com/coldpenguin/esphome. Unfortunately won't have much time this weekend, but I think I know how to get the authentication in place (we can add the aad to the context with ctx I believe, without needing to cast). We could either continue with the 'copied' function, or submit a patch to mbedtls then if it works.

It is going to be necessary to have the auth. I have had two corrupted packets with auth off. This has given me -1100 degrees C and 140% battery.

I believe I read that mbedtls and ccm is used within the expresif code for part of the wpa2 code. So it either mans there should be hope, OR, it means the expresif code is not authenticating packets at all.....

[coldpenguin]

Forgot the config code:

sensor:
  - platform: xiaomi_lywsd03mmc
    bindkey: "8e261731b9bfca36a64831a85b4d73aa"
    mac_address: "A4:C1:38:C6:A3:B8"
    temperature:
      name: "Xiaomi LYWSD03MMC Temperature 1"
    humidity:
      name: "Xiaomi LYWSD03MMC Humidity 1"
    battery_level:
      name: "Xiaomi LYWSD03MMC Battery level 1"
  - platform: xiaomi_lywsd03mmc
    bindkey: "06738f4f36dba46375ba080775e21456"
    mac_address: "A4:C1:38:E5:84:8C"
    temperature:
      name: "Xiaomi LYWSD03MMC Temperature "
    humidity:
      name: "Xiaomi LYWSD03MMC Humidity 2"
    battery_level:
      name: "Xiaomi LYWSD03MMC Battery level 2"
  - platform: xiaomi_lywsd03mmc
    bindkey: "ceadf6d5d83c825a8a09f20eaade37db"
    mac_address: "A4:C1:38:60:42:68"
    temperature:
      name: "Xiaomi LYWSD03MMC Temperature 3"
    humidity:
      name: "Xiaomi LYWSD03MMC Humidity 3"
    battery_level:
      name: "Xiaomi LYWSD03MMC Battery level 3"

[ahpohl]

I believe I read that mbedtls and ccm is used within the expresif code for part of the wpa2 code. So it either mans there should be hope, OR, it means the expresif code is not authenticating packets at all.....

@coldpenguin I got the ccm auth working with mbedtls, but for some reason the tag calculated by the mbedtls_ccm_encrypt_and_tag() function is different from what is expected and given in the packet payload, hence the call to mbedtls_ccm_auth_decrypt() fails. It must be something silly in my code, though. I opened a ticket here and here asking for help on the Arm MBED forum.

BTW: your fork seems to be already in a good shape. great work! Try changing uint8_t aad[8] = {0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11}; to uint8_t aad = {0x11}; and see if auth works.

[coldpenguin]

I might be close :) taglen is actually set to 4,

[15:53:10][C][xiaomi_ble:372]: Token: C6.9D.CA.91 (4)
[15:53:10][C][xiaomi_ble:407]: decrypted message: 04.10.02.40.01 (5)
[15:53:10][C][xiaomi_ble:408]: decrypted token: C6.9D.CA.91.EE.50 (6)

It probably isn't you code at fault. If you are using the mbedtls_ccm_decrypt function, it is very similar to the function I have put in my code. Apart from the fact that the function in my code actually uses an aad now (this is not the same as add). I'll try and get some time to tidy the code and push it tonight (spent last night in a tent)

[ahpohl]

Finally I got the payload decryption working. It was something silly indeed (I used the wrong length for the auth data in mbedtls_ccm_auth_decrypt()).

[22:23:57][D][xiaomi_ble:317]: CCM - Authenticated decryption passed.
[22:23:57][D][xiaomi_ble:327]: Plaintext : 041002E700
[22:23:57][D][xiaomi_ble:187]: Got Xiaomi LYWSD03MMC (A4:C1:38:8C:34:B7):
[22:23:57][D][xiaomi_ble:190]:   Temperature: 23.1°C

TODO: refactor the code to be able to use the bind keys from the config file

[coldpenguin]

Really? From what I can see the code for the mbedtls I have is missing a call to mbedtls_cipher_update_ad

+  if (NULL != aad && aad_len > 0)
+  {
+    mbedtls_cipher_update_ad(&ctx->cipher_ctx, aad, aad_len);
+  }
+

I have pushed my updated code

[ahpohl]

The xiaomi_lywsd03mmc sensor component is now complete and ready for testing. Below is a screenshot taken from Home Assistant (battery level always at 100 % so not so useful to report).

Here are the necessary steps for testing:

git clone https://github.com/ahpohl/esphome
python -m venv venv
source venv/bin/activate
python setup.py install
mkdir Testing; cd Testing
esphome livingroom.yaml run

EDIT: the feature branch xiaomi_lywsd03mmc has been merged the my ahpohl/esphome:dev branch. For testing please use the default dev branch, which contains the "stable" version. The feature branch will continue to contain the "unstable" work in progress version.

The lywsd03mmc sensor part in livingroom.yaml looks like this:

sensor:
  - platform: xiaomi_lywsd03mmc
    mac_address: "A4:C1:38:4E:16:78"
    bindkey: "e9efaa6873f9f9c87a5e75a5f814801c"
    temperature:
      name: "LYWSD03MMC Temperature"
    humidity:
      name: "LYWSD03MMC Humidity"
    battery_level:
      name: "LYWSD03MMC Battery Level"

TODO:

• Test LYWSD03MMC sensors together with other Xiaomi types (I currently do not own any other types, so volunteers are welcome to help)
• Fix travis ci build issue (<mbedtls/ccm.h> not found error)
• Add documentation for new type (esphome-doc)

[coldpenguin]

Ok, I had started a draft PR with my code. I'll remove it

I think you need to check your code where testing the raw[2:3] values, and the types. You'll note in mine that I additionally set the capabilities of the device. This is where your comment

// res.has_value() always returns false when xiaomi_ble::decrypt_xiaomi_payload() // is called from XiaomiLYWSD03MMC::parse_device() // hence I moved the "Got Xiaomi..." device report code to xiaomi_ble::parse_xiaomi()

IF the packet capability is encrypted then it will be false, obviously the code doesn't have the encryption key at that point. If the capability is not encrypted, then the code path should be followed still. However, as I commented on mine, I think that there is code duplication that could be factored out there.

[Joshfindit]

@ahpohl any chance the code can be expanded to get the bind key if it’s not provided in the config?

[vulcan4d]

My guess this hasn't been introduced into the Esphome Hassio code yet? I just tried it, got the bind key using the modded MiHome app but Esphome in Hassio is getting a Platform not found sensor.xiaomi_lywsd03mmc error message.

[1igeza]

* Test LYWSD03MMC sensors together with other Xiaomi types (I currently do not own any other types, so volunteers are welcome to help)

I was testing it few hours and no single reading for miflora. Only:

[04:47:06][D][esp32_ble_tracker:501]: Found device C4:7C:8D:6B:1D:EE RSSI=-81
[04:47:06][D][esp32_ble_tracker:522]:   Address Type: PUBLIC
[04:47:06][D][esp32_ble_tracker:524]:   Name: 'Flower care'
[04:48:04][D][esp32_ble_tracker:501]: Found device 80:EA:CA:88:B1:01 RSSI=-79
[04:48:04][D][esp32_ble_tracker:522]:   Address Type: PUBLIC
[04:48:04][D][esp32_ble_tracker:524]:   Name: 'Flower care'