Xiaomi BLE not asking for bindkey

michalciolek commented 2 years ago

The problem

I want to migrate from Passive BLE Monitor integration to Xiaomi BLE integration. My LYWSD03MMC devices are discovered, when I click Configure they say:

There hasn't been a broadcast from this device in the last minute so we aren't sure if this device uses encryption or not. This may be because the device uses a slow broadcast interval. Confirm to add this device anyway, then the next time a broadcast is received you will be prompted to enter its bindkey if it's needed.

After approve, Device is crreated but without any entity. Prompt for bind key is never showed.

How to configure bindkey for this device?

What version of Home Assistant Core has the issue?

2022.08.5

What was the last working version of Home Assistant Core?

.5

What type of installation are you running?

Home Assistant Core

Integration causing the issue

xiaomi_ble

Link to integration documentation on our website

https://www.home-assistant.io/integrations/xiaomi_ble

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

probot-home-assistant[bot] commented 2 years ago

xiaomi_ble documentation xiaomi_ble source _{^{(message by IssueLinks)}}

probot-home-assistant[bot] commented 2 years ago

Hey there @jc2k, @ernst79, mind taking a look at this issue as it has been labeled with an integration (xiaomi_ble) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

andytuinman3 commented 2 years ago

Same here. Added my sensor about 5 hours ago. Even restarted HA a couple of times. It doesn't ask for a bind key anywhere

Jc2k commented 2 years ago

Have you both used these before, with a bindkey? I have never set up a "new" LYWSD03MMC from scratch; the LYWSD03MMC that has been tested belonged to a beta tester and they reported they got the prompt in < 10 minutes. (Thats the advertising interval of those devices so should never be longer than that). But those were devices that had previously been configured AIUI.

One other option is BlueZ crashed after the device was discovered but before you got its bindkey setup. You could try the 2022.9 beta, which can recover when that happens.

cubertt commented 2 years ago

Same here. Added my sensor. restarted HA a couple of times. It doesn't ask for a bind key anywhere (or i clicked to fast). My Flower thingy from xiaomi works as expected.

Jc2k commented 2 years ago

Same questions as above 😀

cubertt commented 2 years ago

Have you both used these before, with a bindkey?

-->No, i used three complete new devices

One other option is BlueZ crashed after the device was discovered but before you got its bindkey setup. You could try the 2022.9 beta, which can recover when that happens.

-->I am using 2022.8, latest update, but not the beta. i will wait untill 2022.9 stable is there and test it again.

Jc2k commented 2 years ago

Have you given the new devices a bindkey yet?

cubertt commented 2 years ago

no, just read about the bindkey and have no idea what /how do do it.

EDIT: This? https://www.home-assistant.io/integrations/xiaomi_ble#encryption

Jc2k commented 2 years ago

Yep. If they are new devices, they might not have a bind key at all yet. So they can't transmit anything. But the support for your device is entirely passive. If it doesn't transmit anything, it might as well not be there. So it might be if you use the telink flasher and get a bindkey, then that device will maybe suddenly ask for its bindkey in HA. (or it could take up to 10 minutes from when you get the bindkey).

Or the problem could be unrelated. Worth a shot!

michalciolek commented 2 years ago

Have you both used these before, with a bindkey?

I have been using the LYWSD03MMC for over a year, first with the Xiaomi app, then in the HA with the Passive BLE Monitor plugin (token extracted with Xiaomi-cloud-tokens-extractor) so the device is not new. I still have LYWSD03MMC sensors connected to the Passive BLE Monitor in HA and this tool correctly receives all the data (so the devices are transmiting). Maybe it is the problem that I have two tools for these sensors?

One other option is BlueZ crashed after the device was discovered but before you got its bindkey setup. You could try the 2022.9 beta, which can recover when that happens.

I will try tonight.

Maybe HA is asking for bindkey but I can't see it, I don't know where in UI it should appear.

Jc2k commented 2 years ago

Depending on how it is setup, BLE Monitor can interfere with the built in bluetooth support. Active mode is meant to work best, apparently. Also, because it uses packet sniffing rather than going through the whole Linux bluetooth stack it can make it look like things are working when actually your systems BlueZ has crashed. 2022.9 has additional auto-recovery mechanisms to try and cope with bluez and firmware bugs.

The reason to try the beta is so that I could get a fix into the 2022.9 release. It's too late now really. If the fix is big, it will now most likely have to go in 2022.10.

The "reauth" prompt should show up with the other integrations and discoveries. It shows up like a discovery does.

andytuinman3 commented 2 years ago

I just can't get it to work. After a whole lot of restarts it finally asked me for a reauth so I could enter the bind key. Since then only 1 entity was added (temperature) and it stays on 1 spot 24.5 degrees which it was during the reauth. After that it doesn't change. So it still doesn't work.

Jc2k commented 2 years ago

@andytuinman3 it sounds like your bluetooth dongle and version of BlueZ aren't working well together. I bought the exact same bluetooth dongle as like 6 testers had successfully used and i couldn't get it to work without manually poking the dongle with bluetoothctl, and then it would randomly stop working. I tried upgrading bluez, upgrading my OS, backporting things, switching dbus implementations, nothing made it stable. Sometimes it crashed so bad i had to unplug the dongle and plug it back in. Depressingly, from searching around these sorts of Linux bluetooth bugs are quite common. Anyway, I bought a different bluetooth dongle and it's been working fine ever since.

In 2022.9, if the host's bluetooth stack stops reporting data, HA will give it a little poke to try and get it working again. For some setups, this will be enough to gloss over the OS bugs. For some devices that crash frequently or that crash so hard physical re-insertion is required, there is going to be very little we can do.

Unfortunately the default firmware for the LYWSD03MMC only broadcasts once every 10 minutes. That combined with an especially crashy bluetooth dongle are a pain in the butt to debug. So another option with 2022.9 is to use alternative firmware for the LYWSD03MMC. The ATC firmware can broadcast much more frequently, and is supported in 2022.9 by the new BTHome integration. You can even turn off encryption in the firmware config avoiding this whole issue :).

andytuinman3 commented 2 years ago

@Jc2k but why do my miflora and other temperature sensors (also 2 LYWSD03MMC) keep working? And on ATC firmware (which the other 2 are running on) they are not recognized by Xiaomi BLE. I don't know what BThome is but if this is an alternative I will look into it.

Jc2k commented 2 years ago

Are the working sensors under xiaomi_ble?

BTHome is really a new name for one of the formats ATC uses.

Xiaomi_ble doesnt (and won't) support that firmware. But the new BTHome integration in 2022.9 will.

andytuinman3 commented 2 years ago

The miflora is under Xiaomi_ble and still works. I will wait for BTHome for my ATC temp sensors. Thank you for the info

Jc2k commented 2 years ago

@andytuinman3 the LYWSD03MMC that you can't get to work, is it new or did you recently switch it back from ATC to stock firmware? (Even if you are happy to use ATC firmware going forward, i'd like to try and figure out why your LYWSD03MMC doesn't work but it does for other people - so i can help other people with similar issues).

andytuinman3 commented 2 years ago

I switched it back from ATC to stock. Activate it to get the bind key. Added it to HA. Sometimes I get an unknown error sometimes the message about reauth. Waiting doesn't help. Restart a couple of times will help sometimes. But even after adding the bind key it doesn't do jack. I tried getting the log file so I added the debug for xiaomi_ble to my config. But I get no logs (or I just can't find them). Running Home Assistant OS on a VM with a Asus BT500 Bluetooth stick

Ernst79 commented 2 years ago

Just a clarification (not for solving the issue). ATC firmware is able to use different advertising formats, you can set in in the web flashing tool. When you connect to your sensor, you can select one of the different advertising formats, see the differences here.

The Xiaomi BLE integration can work with Mijia standard format with encryption key. It should work like a normal Xiaomi sensor than.
The BTHome integration (available in 2022.9) can work with the BTHome format. Encryption in optional. In the firmware versions < 3.7 this is called HA BLE, in 3,8, it is called BTHome.
The other two formats won't work with HA integrations, only with BLE monitor.

michalciolek commented 2 years ago

I I updated HA to beta version. After that, the sensors appeared in the list and there was already a bindkey field in the dialog! I added one sensor, but unfortunately it only appeared in the integration list. It is missing in in devices and entities:

PS1. The MAC addresses that appear are different than those on the xiaomi servers (and are truncated so I can't see them completely). I had to guess which bindkey matched the given sensor. PS2. IMO the MAC address should also appear in the bindkey entry dialog.

Jc2k commented 2 years ago

It will take up to 10 minutes for the data to appear because of the advertising interval.

michalciolek commented 2 years ago

@Jc2k understand, but I added the sensor 1.5 hours ago and nothing else appears

Jc2k commented 2 years ago

Can you turn on debug logs for homeassistant.components.bluetooth, homeassistsnt.components.xiaomi_ble and xiaomi_ble.

michalciolek commented 2 years ago

I activated the debug before creating an issue. There is nothing in homeassistsnt.components.xiaomi_ble and in homeassistant.components.bluetooth I found:

2022-09-05 19:12:03.345 DEBUG (MainThread) [homeassistant.components.bluetooth.scanner] hci0 (0A:61:11:12:60:04): Scanner watchdog time_since_last_detection: 49.154071586999976
2022-09-05 19:12:33.346 DEBUG (MainThread) [homeassistant.components.bluetooth.scanner] hci0 (0A:61:11:12:60:04): Scanner watchdog time_since_last_detection: 79.155155334
2022-09-05 19:13:03.348 DEBUG (MainThread) [homeassistant.components.bluetooth.scanner] hci0 (0A:61:11:12:60:04): Scanner watchdog time_since_last_detection: 109.15678451900001
2022-09-05 19:13:33.349 DEBUG (MainThread) [homeassistant.components.bluetooth.scanner] hci0 (0A:61:11:12:60:04): Scanner watchdog time_since_last_detection: 139.158051798
2022-09-05 19:13:33.350 INFO (MainThread) [homeassistant.components.bluetooth.scanner] hci0 (0A:61:11:12:60:04): Bluetooth scanner has gone quiet for 110s, restarting
2022-09-05 19:13:33.350 DEBUG (MainThread) [homeassistant.components.bluetooth.scanner] hci0 (0A:61:11:12:60:04): Stopping bluetooth discovery
2022-09-05 19:13:33.368 DEBUG (MainThread) [homeassistant.components.bluetooth.scanner] hci0 (0A:61:11:12:60:04): Starting bluetooth discovery attempt: (1/3)

and a lot of

2022-09-05 19:44:16.782 DEBUG (MainThread) [homeassistant.components.bluetooth.manager] hci0: A4:C1:38:00:09:EB AdvertisementData(local_name='LYWSD03MMC', service_data={'0000fe95-0000-1000-8000-00805f9b34fb': b'0X[\x05\xd5\xeb\t\x008\xc1\xa4\x08'}) connectable: True match: set()

See full log here (A4:C1:38:00:09:EB is configured sensor)

PS HA shows MAC A4:C1:38:00:09:EB as 0009EB... (trunced) - wrong byte order

Ernst79 commented 2 years ago

Can you share your bindkey such that I can check that the data can be encrypted. Note: You can change the bindkey afterwards, by repairing it to MiHome and/or in the PVVX ATC webtool.

michalciolek commented 2 years ago

5d919e209eae851ba10350bbbd427c83

Ernst79 commented 2 years ago

I checked the bindkey, but it works fine. In decodes the data to a temperature of 23.9°C. So the bindkey is not the problem.

Jc2k commented 2 years ago

Just cracked open a fresh LYWSD03MMC. For me it did work. Observations:

If its "fresh" it doesn't seem to broadcast data. Presumably because no bind key. I do get fairly frequent data-less broadcasts. I think this accounts for at least some of the +1's in this issue, but clearly not them all.
If the webtool is "connected" (there will be a bluetooth icon on the LCD) then all transmissions stop. Transmissions start again when I disconnect.
Theres no obvious way to tell from the "empty" beacon that the device isn't bound.
Less than a minute after disconnecting I see a broadcast with a payload. HA shows the reauth prompt immediately.

Screenshot 2022-09-05 at 21 20 29

At this point there is an edge case where I could use the webtool to generate a new bindkey. But HA won't have seen a new transmission, so it will still accept the old bindkey. Worse, it also won't accept the new bindkey until we see a broadcast encrypted with the new bindkey.

(I closed the flasher without copying the bindkey so then I have to make a new bindkey, then wait 10 minutes for HA to let me use it).

When i enter the right bindkey its accepted. After a few minutes, i have a temp and humidity sensor.

My main system is running the official HA beta container on x86_64. I'm using a broadcom based bluetooth device. Theres no VMs anywhere in my stack (i've had trouble with bluetooth and VMs before).

Jc2k commented 2 years ago

@michalciolek Cheers for this, its been very useful.

The log file that you sent seems to cover 19:09 to 21:42. In that time period I only see 31 broadcasts for 0009eb, and actually only 3 of them have data payloads. 19:31 is the last time your system saw data before you sent us the log file.

Your watchdog fires a lot. Between 19:13 and 19:46 it restarted the scanner 12 times. Thats 24 minutes out of 33 where it was radio silent. With one HA instance running, my test (fresh and unused) LYWSD03MMC fires 22 (data-less, annoyingly) broadcasts in a minute. So my watchdog never gets close to firing. And there is just so much other BLE noise, its like a firehose. So i don't understand why yours is firing so much.

At this point in reading the log it's hard not to think that its a hardware problem, and the watchdog is working as intended. That would explain why the beta helped (resetting the bluetooth dongle on a detected fault is new behaviour).

But then after 19:46, no more scanner restarts.... And that doesn't make sense.

So before 19:47, we see quite a few xiaomi_ble compatible devices. But after 19:47, we only see A4:C1:38:F3:17:16. We see one broadcast from that MAC address a minute. That is enough to keep the watchdog happy. So no more restarts.

But we don't see any data for 0009eb either. So its like all the other Xiaomi devices suddenly got powered down or went out of range of bluetooth on your HA box.

Is there a lot of metal near your bluetooth adaptor? Lots of cables maybe? USB3?

Jc2k commented 2 years ago

It looks like there were Apple devices (manufacturer 76) and Samsung (117) visible to your bluetooth adaptor until ~19:47 too, and then they disappeared.

michalciolek commented 2 years ago

This sensor is the farthest from the BLE receiver. The BLE receiver is plugged directly into the server's USB port, next to it are the ZigBee and RTL433 receivers. Probably causing the interference. With the current Passive BLE plugin it works well enough for me - the measurement frequency is sufficient (blue). The closer sensor has a higher frequency of measurements (purple).

Unfortunately, the device and entity still do not appeared with the new Xiaomi BLE integration.

Jc2k commented 2 years ago

The logs I'm reading are unfortunately from the layer below the xiaomi ble integration and they tell me that HA just can't see very much information from Bluetooth at all, at least when using BlueZ (which is the user space part of the official Linux Bluetooth stack).

Passive BLE Monitor is more like plucking raw Bluetooth data out of the air - you could say that it's somewhat like rtl433 in how it works. BlueZ apparently needs to do extra stuff that it doesn't. Maybe a better example is that it's more like trying to listen to HTTP traffic with tcpdump instead of going through the Linux TCP/IP stack.

As it stands if bluez can't see these devices you can't use any of the new Bluetooth integrations. There isn't a HA fix that can increase the range of BlueZ, so we are just stuck.

You could use a usb extension to get your dongle away from the interference sources (I do this). Also 2022.9 has support for remote dongles (using something like usbip to "mount" a dongle on a remote computer/raspi/etc over your home Ethernet or WiFi. It also has support for using ESP32 chips as remote Bluetooth listeners.

michalciolek commented 2 years ago

OK, range and interference in this case, we can ignore, because @Ernst79 was able to read the temperature from the logs. The question is why HA didn't do this (no entity).

Passive BLE plugin works great for me and I could stay with it, but there is information that support will be dropped so I wanted to check the integration which replace it.

Toninght I will connect the BLE receiver via the cable and see if it will improve anything.

Jc2k commented 2 years ago

There were only 3 valid packets in the logs, and not enough information in the logs to conclusively say what state HA was in when they were received. For example, if they were received during a HA restart xiaomi_ble might not have been fully loaded. Or they could have been received and logged before the bindkey was added and applied. There are probably other scenarios like that.

Ernst79 commented 2 years ago

Yes, I only checked the first message. But as this was the first message where xiaomi-ble was recognized, you probably didn't had the bind key entered yet.

2022-09-05 19:09:59.340 DEBUG (MainThread) [homeassistant.components.bluetooth.manager] hci0: A4:C1:38:00:09:EB AdvertisementData(local_name='LYWSD03MMC', service_data={'0000fe95-0000-1000-8000-00805f9b34fb': b'XX[\x05\xc5\xeb\t\x008\xc1\xa4OZ\xdaBH^\x02\x00Hq\x1e\xd0'}) connectable: True match: {'xiaomi_ble'}

You mention that you also use BLE monitor. Just a double check (forgive me if you already checked). But do you use these both at the same time? If, so, make sure you enable active scan in BLE monitor, to use the same scanning mode. And if that doesn't work, try to disable BLE monitor.

michalciolek commented 2 years ago

I use this two integration at the same time. Active scanning in BLE monitor is disabled.

Active scan it means that ble monitor will connect to device to read data?

Tonight I will enable active scan or disable ble monitor and check if it help.

Jc2k commented 2 years ago

Active scanning is different to connecting and reading values. It's still just listening to broadcasts made by the sensors. But we also send a broadcast of our own asking the sensors to transmit more often. This is much power efficient than an actual connection.

The new HA integration defaults to active because bluez support for anything else is experimental. So I'm if BLEM doesn't do the same it can in theory cause conflicts. It's definitely worth ruling that out as the cause.

I'm still suspicious of your log data though. Why is there 2 hours where only a single device is visible. If BLEM and HA were fighting and restarting their scanners constantly I'd expect the firehose of Bluetooth data to be quieter sure. But I'd expect to see a random sample of data. Not 1 device and not such a regular cadence.

But who knows with these things 🥸

Ernst79 commented 2 years ago

It won't connect, but it will send a message "please send me more data". Normally, this isn't needed for Xiaomi devices, but the Bluetooth implementation in HA is using active scan (in 2022.8). BLE monitor is using passing scan. So, your Bluetooth dongle/radio gets confused, should it use active or passive scan. Normally, BLE monitor wins this battle and it will use passive scan, which results in non-working HA Bluetooth integrations.

The solution is to use active scan in both integrations, or using a different dongle for each integration.

I'm not sure about 2022.9. I thought @bdraco made a PR that the HA bluetooth integrations can also use passive scan, but not 100% sure. @Jc2k do you know if that made it to 2022.9? If so, it means that you need to change it back in 2022.9.

Jc2k commented 2 years ago

Passive scan is experimental in bluez. The feature flag isn't turned on in HAOS AIUI so it's a no go there.

But if you have a suitably manually configured host OS yes passive is possible.

michalciolek commented 2 years ago

@Jc2k thank for your explanation. I turned on active scan yesterday at 20.00 and this seems has resolved problems. After that sensor receives entities. Below is comparison BLE monitor (bleu) Xiaomi BLE integration(purple) - more readouts than BLE monitor

I added also another sensor with: There hasn't been a broadcast from this device in the last minute so we aren't sure if this device uses encryption or not. This may be because the device uses a slow broadcast interval. Confirm to add this device anyway, then the next time a broadcast is received you will be prompted to enter its bindkey if it's needed. after few minutes I have received notification to input bindkey. So everything's seems to work well. Thank you!

Although it works, I am considering going back to the BLE monitor. Active scanning does not convince me (apparently it is supposed to use up sensor batteries more).

Jc2k commented 2 years ago

That's good to hear!

There are superstitions around active mode. I believe that when it was tested the actual difference was negligible. Although it depends on the devices. Disclaimer - I have not tested either.

Anecdotally my plant sensors are still showing a full battery after 2 months of active scanning, though I'm told the sensor is rubbish.

Also remember that if any device is doing active scanning all devices are doing in active mode, regardless of which integration you are using. So even if you removed every home assistant integration that was doing Bluetooth, any device in your house (or neighbours!) could induce the same scanning mode.

Also remember it's not just xiamoi_ble you need to avoid going forwards, it's pretty much any Bluetooth integration in HA.

michalciolek commented 2 years ago

You convinced me. I will migrate the remaining sensors and I will test. I have no more questions, so I'm closing the issue.

Regards!

cubertt commented 2 years ago

Active scanning is a feature of the beta version of the Xiaomi BLE integration?

Jc2k commented 2 years ago

No. Active scanning is the only kind possible on Linux systems running bluez (like ha) without turning on experimental features.

cubertt commented 2 years ago

It worked now after:

Adding the sensor to xiaomi app on my phone
updating hass.io to Septemper release
deleting the sensor in integrations -- xiaomi ble
Waiting until the sensor gets recognized again (i removed the batterie to restart the sensor)
Some info appeared like here: https://github.com/home-assistant/core/issues/77681#issuecomment-1239737912
i just said ok,
Waiting again, then it asked for the ble key
Getting key via the tool mentioned in the xiaomi ble docs
adding the key

et voila (i hope so, now i have two entities, temp and signal strength. Is this enough, i am missing at least humidity?)

Jc2k commented 2 years ago

Sometimes it can take a while for all the sensors to show up - especially if they are at the edge of the reception zone.

home-assistant / core