Scanner disconnects ble connection that runs in a separate thread

[SebasScript]

Hi, I am subscribing to a BLE notification and streaming the data from the BLE device notification in a python thread using
p.waitForNotifications as described in the documentation. If i run the scanner in a separate thread using scanner = btle.Scanner().withDelegate(ScanDelegate()) The connection in the first thread is lost after the scanner is executed with this error:

File "/home/pi/apps/aih/lib/ble.py", line 148, in readNotifications peripherals[device_name].waitForNotifications(1.0) File "/usr/local/lib/python3.5/dist-packages/bluepy/btle.py", line 560, in waitForNotifications resp = self._getResp(['ntfy','ind'], timeout) File "/usr/local/lib/python3.5/dist-packages/bluepy/btle.py", line 407, in _getResp resp = self._waitResp(wantType + ['ntfy', 'ind'], timeout) File "/usr/local/lib/python3.5/dist-packages/bluepy/btle.py", line 362, in _waitResp raise BTLEDisconnectError("Device disconnected", resp) bluepy.btle.BTLEDisconnectError: Device disconnected

Is this a limitation of the library (i can't run the scanner while simultaneously receiving data through notifications) or is there an error on my site?

[SebasScript]

350

This pull request is related to my issue

[johnfwhitmore]

I'm the author of that pull request you mentioned, #350, that's been a while ago, but I still have to use that one line change.

This library might not have been designed for handling multiple devices, so my change might not be well thought out but it worked for me. I've never been contacted about that proposed change so I'm not sure that there's any activity on this project. There's definitely interest in BLE but I'm finding the whole stack very unstable on Ubuntu 16.04. If I wasn't fire fighting I might be able to look at this in the cold light of day, but at present no time.

[SebasScript]

Thanks for the reply. The library works with multiple devices. I did connect two Ble devices, subscribed and received notifications using both Threading and Multiprocessing. Good to know about Ubuntu instability. It works pretty well on Raspbian

[johnfwhitmore]

I should say that the Ubuntu thing might well be me, as in something I'm doing wrong. I have had up to 7 devices connected, but then get a HCI error in syslog. When I say up to 7 at present 5 is my limit. I'm trying to find a reason for that but documentation is hard to come by. At present I'm down in the bluez code which is distributed with bluepy to find a reason for the limit. At present I have a number which means nothing.

On the multitasking I do a regular scan for new devices, of a particular type, to connect them. If you're scanning you can't send a command to any of the connected devices, but have to buffer that and send it after the scan is complete. That's understandable as there's only one radio interface. Still I'm writing code to handle that and perhaps that should be at a lower level. Must try see what happens if I have two python programs trying to scan and connect to different devices. That'd test it ;)

[SebasScript]

You do the scanning while buffering the commands for the devices with the one line modification you made? As i understand from your pull request the scan mode automatically disconnects all other devices? I could pause my waitForNotifications processes before scanning and reactivate them after. I thought i tried that but i don't remember ;).

Just a thought on your device connection limit. Maybe your bluetooth module doesn't support more than 5 simultaneously connected devices? I read that a lot of modules place an upper limit on max connected peripherals.

[mikimer]

@johnfwhitmore I had a problem last year running bluepy on Ubuntu too. I don't remember all the details, but here's the error I received and how we got the code to (kind of) run in the end. I hope this helps!

[sterwen]

Opened an issue #370 that looks similar to this one. I have investigated the problem and this not lnked to multi-threading. This is just that if the HCI layer stops the scan, there is no message sent back to the Python. I will make a private correction as it looks that all pull request are stalled since nearly a year. This is a problem in bluepy-helper.

[Firefox2005]

I will make a private correction as it looks that all pull request are stalled since nearly a year. This is a problem in bluepy-helper.

Would you mind sharing your changes? I'm facing the same problem and could need some help.

[sterwen]

Hello, Finally I did not made any changes in the bluepy-helper for that, but improved slightly the robustness on the python library. The problem seems to come form the Bluetooth chip itself, and we are still investigating the problem with the chip vendor and we will also start to test new chips (Nordic). I am have no problem to push my changes, but it seems that no pull request have been processed since a while.

[choonkiatlee]

Hi!

I experienced the same issue as @shantiscript where my BLE devices would be disconnected when running the scanner in a separate thread.

A workaround that worked for me was to instead run the scanner in a separate process using the multiprocessing library in python instead. This creates a separate copy of the bluepy-helper program, allowing the scanner class to modify device states without affecting the currently connected Peripheral devices. Hope that this helps!

from multiprocessing import Process, Event

class BLEScanner:
    def __init__(self):
        self.scanner = Scanner().withDelegate(ScanDelegate())

        self.stop_event = Event()

    def start(self):

        self.stop_event.clear()

        self.process = Process(target=self.scan, args = ())
        self.process.start()
        return self

    def scan(self):
        while True:

            if self.stop_event.is_set():
                return

            self.devices = self.scanner.scan(5, passive=True)

    def stop(self):
        self.stop_event.set()  

Usage: BLEScanner().start()

[peter9477]

For what it's worth, this is easily reproducible with just bluepy-helper and blescan. If you run bluepy-helper manually, connect to a device (i.e. "conn aa:bb:cc:11:22:33 random hci0"), then run blescan, the scan results will also show up in the bluepy-helper output, and shortly after that scan starts you'll also see a "rsp=$stat state=$disc" message. You're not actually disconnected, however, as you can tell if you have enabled a notify. The notifies will continue arriving even after the "$disc".

I'm not sure what e.g. gatttool does differently since scan results don't show up there and you don't get disconnected, so I'm guessing it's something with incorrect filtering of the HCI messages at some level. Perhaps bluepy-helper should be filtering out messages meant for other clients, and isn't? I have no idea yet... I'm still just getting my feet wet on this stuff.

[peter9477]

Given my last comment and further experimentation, I don't understand how running a separate process would help @choonkiatlee as mentioned in his comment. The issue is ultimately that the scan results are going to multiple processes, so with the current bluepy-helper code it should still fail.

By the way, I tried adding a hack to let me force "conn_state = STATE_CONNECTED" and confirmed that, if I run that after being told I'm disconnected (after another process runs a scan and kills me), then I can continue with the existing connection with no problems.

So this isn't a "real" problem affecting the actual connection, which is still valid. It could be handled in bluepy-helper either with a workaround or possibly by eliminating those bogus scan results.

The workaround would probably be to know when we haven't asked to start scanning, and ignore the unexpected/unwanted scan results. Currently whenever it sees a MGMT_EV_DISCOVERING event, it immediately sets the state to SCANNING, and that's what triggers it to change to DISCONNECTED when the scan ends. I guess you could even just check if state is CONNECTING or CONNECTED before doing anything with that event. If it is, just drop it since it wasn't expected.

Proof of concept: modify function mgmt_scanning to have this:

    if (conn_state != STATE_CONNECTING && conn_state != STATE_CONNECTED) {
        DBG("Scanning (0x%x): %s", ev->type, ev->discovering? "started" : "ended");

        set_state(ev->discovering ? STATE_SCANNING : STATE_DISCONNECTED);
    }

Note that this prevents you from initiating a scan while connected. I believe, though, that even though that appears possible with the unmodified bluepy-helper, it would screw things up because it would set your state to DISCONNECTED at the end, even though you're not. Probably unrecoverable at that point without exiting and restarting the process.

(For anyone interested, I made that change and ran it, and currently have two processes connected happily to two devices while a third is running a continuous passive scan, with no apparent problems.)