Connection aborts after firmware update

Markkuuss commented 5 years ago

Since I updated my Raspberry (Linux 4.19.42-v7+ #1219; newest released kernel), the Miflora sensor is no longer accessible after some time. In the beginning, when I started the service, it works fine for some time, but after a while the Miflora sensor is no longer accessible. It looks similar to what @aymeric106 described in #80.

The sensor no longer returns any values. Even a restart of the service or even of the complete system does not result in the connection being restored.

By rebooting several times, connecting to the Android App, removing the battery, etc. I made it work again somehow. But it is not an unique problem. The battery is 99% full and it has happened several times after the update.

I still have no idea what it could be. Do any of you have a similar problem after upgrading the kernel?

Markkuuss commented 5 years ago

I just noticed that if the sensor is not reachable again and I execute the following command, it returns an error message:

sudo hcitool lescan

LE Scan ...
62:2B:4F:27:9D:6C (unknown)
62:2B:4F:27:9D:6C (unknown)
C4:7C:8D:67:3A:CC (unknown)
C4:7C:8D:67:3A:CC Flower care
Disable scan failed: Input/output error

netzwerkgoettin commented 5 years ago

Hi @Markkuuss,

I can confirm this behaviour, same kernel. I also rebooted, restarted and so on, but no difference – I only got this:

[2019-06-17 00:39:43] Initial connection to Mi Flora sensor "Konferenzpflanze" (C4:7C:8D:67:4C:9E) failed.

[2019-06-17 00:39:43] Announcing Mi Flora devices to MQTT broker for auto-discovery ...
[2019-06-17 00:39:43] Retrieving data from sensor "Konferenzpflanze" ...
[2019-06-17 00:40:13] Retrying ...
[2019-06-17 00:40:44] Failed to retrieve data from Mi Flora sensor "Konferenzpflanze" (C4:7C:8D:67:4C:9E), success rate: 0%

[2019-06-17 00:40:44] Sleeping (300 seconds) ...

Interesting: I have a running Home Assistant instance, miflora configured as a sensor; it stopped working, too – HA only reports messages like these:

2019-06-17 00:15:35 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.orchidee_battery is taking over 10 seconds
2019-06-17 00:16:05 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.orchidee_conductivity is taking over 10 seconds
2019-06-17 00:35:36 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.orchidee_moisture is taking over 10 seconds

The iPhone App Flower Care detects this miflora device, connects it and retrieves the data! So what – at least for the moment – helped was this: placing miflora within spitting distance to RPi. I mean: not more than 10 or perhaps up to 20cm. So I finally got it working again (both miflora-mqtt-daemon and Home Assistant instance!).

Adding sensor to device list and testing connection ...
Name:          "Konferenzpflanze"
Internal name: "Konferenzpflanze"
Device name:   "Flower care"
MAC address:   C4:7C:8D:67:4C:9E
Firmware:      3.1.9
[2019-06-17 00:46:01] Initial connection to Mi Flora sensor "Konferenzpflanze" (C4:7C:8D:67:4C:9E) successful

[2019-06-17 00:46:01] Announcing Mi Flora devices to MQTT broker for auto-discovery ...
[2019-06-17 00:46:01] Retrieving data from sensor "Konferenzpflanze" ...
[2019-06-17 00:46:01] Result: {"moisture": 0, "light": 110, "battery": 100, "conductivity": 112, "temperature": 29.3}
[2019-06-17 00:46:01] publishing to mqtt topic "homeassistant/sensor/konferenzpflanze/state"

[2019-06-17 00:46:02] Sleeping (300 seconds) ...

That's really crazy. Does distance make a difference for you, too? I just started to search for 4.19.42-v7+, bluetooth and problem and there seem to be some... so probably it's a more general problem. But it's way too late now for further researching... :sleeping:

Cheers, Marianne

ThomDietrich commented 5 years ago

Hey guys, Hey Marianna! This is crazy! In general be aware that updating the RPi firmware (rpi-update) is not generally a good idea. It's recommended to stay with the default. Could you try reverting? I believe these instructions might help: https://www.raspberrypi.org/documentation/linux/kernel/updating.md

Markkuuss commented 5 years ago

I have changed the firmware of my Raspberry back to the kernel with the last kernel version 4.14: https://github.com/Hexxeh/rpi-firmware/commit/a08ece3d48c3c40bf1b501772af9933249c11c5b

So it works again.

The command for this is: sudo rpi-update a08ece3

@ThomDietrich: But what happens if I download the current Raspbian Image? Then the default firmware has a kernel higher than 4.14 and Miflora won't work anymore?

Markkuuss commented 5 years ago

I tested it again yesterday. With the current stable kernel 1.20190517, which comes with the standard Raspbian update/upgrade process (sudo apt-get dist-upgrade), miflora-mqtt-daemon no longer works.

The previous stable kernel version 1.20190401 (https://github.com/raspberrypi/firmware/releases/tag/1.20190401) is available with a downgrade via rpi-update: sudo BRANCH=stable rpi-update 45a2e771e7272781e62ae92322734c2b90e0268a So it works fine again.

@sysadmama: I didn't notice any difference in the distance. In the beginning it runs with the current kernel at exactly the same distance as with the old kernel. But if it runs longer, after a certain time no connection can be established. This is also reproduceable.

krikk commented 5 years ago

it looks like this is the problem why it is not working on my brand new raspberry pi 4...

Markkuuss commented 5 years ago

it looks like this is the problem why it is not working on my brand new raspberry pi 4...

Probably that's the problem. Post your Linux version - the output of the command "uname -a".

krikk commented 5 years ago

i don't understand why, but it started working again... so forget my post...

[18:55:25] root@raspi4:/opt/miflora-mqtt-daemon# uname -a Linux raspi3plus 4.19.50-v7l+ #895 SMP Thu Jun 20 16:03:42 BST 2019 armv7l GNU/Linux

milkpirate commented 5 years ago

I had similar problem with not working connections or working for a while an then not anymore. Since the downgrade (like @Markkuuss) suggested, everthing seems fine. Will do further testing...

lionhe1966 commented 5 years ago

I am experiencing the same problem on a raspberry pi v3 with Stretch since a long time, but it seems to me that my kernel is older than the one reported here. My kernel is

Linux raspberrypi2 4.14.79-v7+ #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l GNU/Linux

I implemented a rule in OpenHAB (apt-get version 2.4 installed on raspbian stretch) that restarts bluetooth and miflora services. It happens more ore less daily and it seems related to the weakness of the signal (with low batteries it occurs more often?)

I updated right now to the version suggested a few post above, and in my case it does not seem a downgrade, rather it is an upgrade.

Linux raspberrypi2 4.14.98-v7+ #1200 SMP Tue Feb 12 20:27:48 GMT 2019 armv7l GNU/Linux

I will see how it proceed and will report back (Edit:So far, after 4 days, no restart of bluetooth service was required)

mvalla commented 5 years ago

in my case the daemon works on a RaPi with Raspbian (no OH installed) and kernel: Linux raspberrypi 4.19.57-v7+ #1244 SMP Thu Jul 4 18:45:25 BST 2019 armv7l GNU/Linux Success rate 100%.

the daemon does not work if activated on another RaPi with openhabian and kernel: Linux openhab 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l GNU/Linux

the 2 RaPis (both are model 3b+) are positioned close to each other so position is not part of the problem. I even tested switching the SD card of the two RaPi, and the behaviour becomes switched, so it does not depend on the hardware either.

Markkuuss commented 4 years ago

Any news here? Has anyone made it stable with the latest firmware?

I'm afraid my attempts were all negative. After some time no more data will be read.

cropab commented 4 years ago

Any news here? Has anyone made it stable with the latest firmware?

I'm afraid my attempts were all negative. After some time no more data will be read.

Tried it on Pi 0, 3 and 4. Same issue. Its even the same in Node Red.

a-x- commented 4 years ago

I have an error too. I installed this library first time

raspberry pi zero w (with ble), uname -a Linux 4.19.75+ #1270 Tue Sep 24 18:38:54 BST 2019 armv6l GNU/Linux

cat /etc/*-release PRETTY_NAME="Raspbian GNU/Linux 10 (buster)" NAME="Raspbian GNU/Linux"

xiaomi mi flower care firmware: 3.2.2

bluez is the newest version (5.50-1.2~deb10u1+rpt1).

...
Adding sensor to device list and testing connection ...
Name:          "flower_3"
[2020-05-02 19:19:45] Initial connection to Mi Flora sensor "flower_3" (C4:7C:8D:6A:65:12) failed due to exception: 

[2020-05-02 19:19:45] Announcing Mi Flora devices to MQTT broker for auto-discovery ...
Traceback (most recent call last):
  File "miflora-mqtt-daemon.py", line 293, in <module>
    'sw_version': flora['firmware']
KeyError: 'firmware'

bluetooth daemon has some warnings too: sap-server: Operation not permitted (1)

``` ● bluetooth.service - Bluetooth service Loaded: loaded (/lib/systemd/system/bluetooth.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2020-05-02 18:52:33 MSK; 55min ago Docs: man:bluetoothd(8) Main PID: 366 (bluetoothd) Status: "Running" Memory: 3.0M CGroup: /system.slice/bluetooth.service └─366 /usr/lib/bluetooth/bluetoothd May 02 18:52:33 orpik2 systemd[1]: Starting Bluetooth service... May 02 18:52:33 orpik2 bluetoothd[366]: Bluetooth daemon 5.50 May 02 18:52:33 orpik2 bluetoothd[366]: Starting SDP server May 02 18:52:33 orpik2 systemd[1]: Started Bluetooth service. May 02 18:52:33 orpik2 bluetoothd[366]: Bluetooth management interface 1.14 initialized May 02 18:52:34 orpik2 bluetoothd[366]: Sap driver initialization failed. May 02 18:52:34 orpik2 bluetoothd[366]: sap-server: Operation not permitted (1) ```

——

little more debug info (print(flora_poller.__dict__)):

{'_mac': 'C4:7C:8D:6A:65:12', '_bt_interface': <btlewrap.base.BluetoothInterface object at 0xb60c0bb0>, '_cache': None, '_cache_timeout': datetime.timedelta(seconds=299), '_last_read': None, '_fw_last_read': None, 'retries': 3, 'ble_timeout': 10, 'lock': <unlocked _thread.lock object at 0xb60ce140>, '_firmware_version': None, 'battery': None}

sudo hcitool lescan
LE Scan ...
C4:7C:8D:6A:65:12 (unknown)
C4:7C:8D:6A:65:12 Flower care

more exceptions (I commented some except branches and revealed some exception)

```Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/btlewrap/bluepy.py", line 27, in _func_wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.7/dist-packages/btlewrap/bluepy.py", line 56, in connect self._peripheral = Peripheral(mac, iface=iface, addrType=self.address_type) File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 391, in __init__ self._connect(deviceAddr, addrType, iface) File "/usr/local/lib/python3.7/dist-packages/bluepy/btle.py", line 439, in _connect raise BTLEDisconnectError("Failed to connect to peripheral %s, addr type: %s" % (addr, addrType), rsp) bluepy.btle.BTLEDisconnectError: Failed to connect to peripheral C4:7C:8D:6A:66:FC, addr type: public The above exception was the direct cause of the following exception: Traceback (most recent call last): File "miflora-mqtt-daemon.py", line 212, in flora_poller.fill_cache() File "/usr/local/lib/python3.7/dist-packages/miflora/miflora_poller.py", line 85, in fill_cache firmware_version = self.firmware_version() File "/usr/local/lib/python3.7/dist-packages/miflora/miflora_poller.py", line 127, in firmware_version with self._bt_interface.connect(self._mac) as connection: File "/usr/local/lib/python3.7/dist-packages/btlewrap/base.py", line 47, in __enter__ self._backend.connect(self._mac) File "/usr/local/lib/python3.7/dist-packages/btlewrap/bluepy.py", line 33, in _func_wrapper raise BluetoothBackendException() from last_error btlewrap.base.BluetoothBackendException ```

but it’s still not clear

I also tried to pair with xiaomi flower care devices via bluetoothctl:

pair C4:7C:8D:6A:66:F2
Attempting to pair with C4:7C:8D:6A:66:F2
[CHG] Device C4:7C:8D:6A:66:F2 Connected: yes
Failed to pair: org.bluez.Error.AuthenticationFailed
[CHG] Device C4:7C:8D:6A:66:F2 Connected: no

I keep devices nearby

a-x- commented 4 years ago

UPD. I found my mistake. I must run it with sudo!

Make errors clear, please! It took many hours of my life. Good example: another miflower api project: https://github.com/vrachieru/xiaomi-flower-care-api. It said I must run with sudo. When I tried it here too, it started works. Also, add please note about sudo into readme.

Any way, big thank you for your work!

ThomDietrich commented 4 years ago

Hey @a-x-

thanks for the exhaustive analysis. Let me just clarify that root privileges are not needed for the daemon. Also see the service definition: https://github.com/ThomDietrich/miflora-mqtt-daemon/blob/master/template.service#L8 I wonder if there were changes in buster leading to missing permissions, e.g. users not being added to the bluetooth group. Would you be willing to check this?

Quick note: The KeyError: 'firmware' was just fixed in https://github.com/ThomDietrich/miflora-mqtt-daemon/pull/118 but isn't really related to your issue.

Make errors clear, please!

I thought we already do that. Which error message is missing in your opinion? A pull request would be highly appreciated!

Markkuuss commented 4 years ago

I had similar problem with not working connections or working for a while an then not anymore. Since the downgrade (like @Markkuuss) suggested, everthing seems fine. Will do further testing...

Were you able to find anything else in the meantime, @cropab and @milkpirate ?

I'm currently at kernel version 4.19.66-v7+ #1253. Unfortunately it doesn't work after a while. Then I have to do a reset every time: sudo hciconfig hci0 reset

Markkuuss commented 4 years ago

I think I have now found the solution. With the firmware, which is currently distributed as a stable with Raspbian Stretch (1.20190819 / 4.19.66), the Bluetooth does not work correctly on my Raspberry Pi 3b+. The problem existed since the first update to 4.19.

I have now updated to the last available version 4.19.118 (1.20200512: kernel: Latest 4.19). Since then miflora runs stable again, at least it runs 24h without problems. This was unthinkable before.

The command for the update to the last 4.19 is sudo rpi-update 866751bfd023e72bd96a8225cf567e03c334ecc4

ThomDietrich / miflora-mqtt-daemon

Connection aborts after firmware update #83