Change of sensor behaviour after updating to 4.5

[yannpub]

Here are the readings I get after update to 4.5 :

Device is now known as "Temperature/Humidity sensor by Xiaomi " with 4.5/4.6, whereas it was considered as "BTHome sensor by Xiaomi "sensor with 4.4 My device which is still on 4.4 isn't impacted, only the 2 devices I upgraded.

[pvvx]

Since fw version 4.5, the BLE advertising protocol has been replaced from BTHome version 1 (deprecated as of Nov '22) to BTHome version 2.

[yannpub]

I saw in the changelog, but change of protocol shouldn't change the readings of the values. Or is there an additional change to be done on the HA side to get the readings properly after the migration to the new protocol?

[cpuks]

What about removing and adding device? {Maybe extra restart of HA in-between)

[pvvx]

Nothing changes in Home Assistant with standard integrations (BTHome, Passive BLE Monitor). Switching occurs automatically...

[yannpub]

In your screenshot, sensor type is identified as 'BTHome', which was the behaviour I had with firmware 4.4, and which is providing proper reading. After the upgrade the sensor type changes to "Temperature/Humidity sensor", which I assume is what triggers a different interpretation of the values. I am on HA 2023.11, using BTHome integration.

[pvvx]

For example, I reprogrammed a thermometer operating in the “BLE Long Range” mode from version 4.2 to 4.6. Nothing has changed in the "BTHome" integration

Nothing has changed, except for temporary readings of changes in humidity that occurred from picking up the thermometer, and a temporary loss of transmission, since the thermometer switched to BLE Legacy for firmware. Then switched back to "BLE Long Range". This is specifically the most difficult case.

In your screenshot, sensor type is identified as 'BTHome'

'BTHome V2'

[yannpub]

It seems the issue was coming from my Intel Bluetooth module, which is known to have some issues with the current 6.1 kernel : https://github.com/home-assistant/operating-system/issues/2535#issuecomment-1784219788 All my devices are now updated on FW4.6, BTHome v2 and have proper readings in HA.

Shutdown/restart of the server was the solution to get stable readings. Still doesn't understand though why the firmware update triggered this lower layer bluetooth issue and different readings, but problem solved !

[cpuks]

Cool cheers for clearing this out - I'll upgrade to 4.6 then - I knew you need to restart ;-)

[yannpub]

Cool cheers for clearing this out - I'll upgrade to 4.6 then - I knew you need to restart ;-)

Physical restart of the machine was necessary - restart of HA didn't do the trick.

[chatziko]

This is one of the strangest issues I've seen in a while. In my system I run HA OS on a VM on top of Proxmox running on a NUC. I do not use the NUC's bluetooth adapter (I don't even pass it through to the VM), but I rely on two ESP32 bluetooth proxies instead.

• I first restarted the proxies, nothing changed.
• Then I restarted Home Assistant, again nothing.
• Then I rebooted the HA OS VM, still nothing!
• Finally, I rebooted the whole NUC (without much hope) and magically the problem is gone!

How is it possible to require a full host restart to solve an issue that is not even related to the host's hardware? Bluez magic...

In any case I'm super happy the issue is gone, thanks for this awesome project :smile:

[chatziko]

I declared victory too fast, the problem is still hapenning. It's much rarer now, but it's there.

Would it make sense to try v4.6, or should I better downgrade to v4.3? Anything else I could try?

[pvvx]

1. If you are using a BT adapter - delete all files /var/lib/bluetooth/
2. Bluetooth ESP32 often glitches. Then the only option is not to use ESP.
3. Find where old data is stuck.

@chatziko - Why is the data shown at such unequal intervals? Losses in reception - a lot of dropouts? I have more than 30 BLE devices to one USB-BT. At the same time, there are more than 40 BLEs on the air nearby, a dozen of which broadcast BLE advertisements every 50 ms. There should have been a lot of reception collisions, but they are not visible on the graphs.

[chatziko]

For anyone having the same issue while using bluetooth proxies, I think I finally solved the issue by doing the following:

• stop HA
• delete /config/.storage/bluetooth.remote_scanners
• start home assistant again

Not sure what exactly this file is doing. After the restart it was not recreated right away (although it did reappear after some time) and the issue seems solved (although I was trying lots of things so I can't be 100% sure about what exactly was causing it).

[chatziko]

@chatziko - Why is the data shown at such unequal intervals? Losses in reception - a lot of dropouts?

I guess the difference is that you are using Passive BLE Monitor while I am using the native BTHome integration.

In your plot it looks like you are getting exactly 1 sample per minute. Based on the 60 seconds "period to use for averaging" setting, I would guess that Passive BLE Monitor creates these regular samples, independently from when exactly the raw data arrive!

In my case, on the other hand, I think the sensor reports the raw data. I set advertising interval to 6000 msecs, so under perfect conditions I suppose I should have 10 samples per minute, but many are lost so in the plots I see between 1 and 3 samples per minute.

[pvvx]

In your plot it looks like you are getting exactly 1 sample per minute.

Yes. New data arrives every 10 seconds and is rounded over a period of 60 seconds. If the data for the period is not updated, the previous value will remain.

In "BTHome Integration" the schedule is more disorderly, but there are no gaps of more than 1 minute. "Advertising interval up to 6000 ms" is the transmission duplication time. The sensor is polled and data updated after a user-specified number of transmission intervals. When installed by default, these are 4 such intervals. 6*4 = 24 second. - As a result, every 24 seconds the values on the graph should change. In this case, the values were transmitted 4 times over 3 channels. That is 12 times. Even if the dropout rate is 75%, the graph will still update every 24 seconds. But the number of lost, not received RF packets in your system is extremely high.

[mhoran]

@chatziko your steps above also worked for me in resolving this issue. I have to use the ESPHome Bluetooth relay as whatever combination of BlueZ / drivers / etc. does not work with my setup (Home Assistant does not see any of my Bluetooth thermometers, even when using a high performance Bluetooth adapter.) But the ESPHome was working great for me with the 4.3 firmware; it stopped working with 4.5.

I think this is pretty clearly a Home Assistant issue, but no idea what it could be. Presumably something with migrating a BTHome device from v1 to v2. I tried just about everything else -- rebooting the ESPHome device, deleting all the thermometers, restarting Home Assistant, resetting the thermometers, etc. The only thing that worked was deleting the above file and restarting Home Assistant. Presumably that does some sort of re-registration of the ESPHome with Home Assistant.

What I was seeing is that normal values were coming from the thermometer regularly (seemingly no dropped packets.) But then every once in a while there would be a spurious temperature reading 72.99. This would cause the graph to jump but then a proper value would replace it seconds later. So something must be getting cached in Home Assistant, overriding real temperature values. Also, when I saw these readings, I believe the timestamp was "1 second from now", which I thought was a bit odd.