Bluezero retrospective

[ukBaz]

I feel this repository/library is at a watershed moment. As there are a number people showing some level of interest I wanted to inform and hopefully get feedback on where I think we need to go next.

Building BlueZ from source

Having to build BlueZ from source is putting people off. Even though the steps are documented it doesn’t fit with the desired goal of making Bluetooth accessible. The main Linux platforms we have been using have OSes based on Debian and Debian’s next version (Stretch) is going to have BlueZ 5.43 as the default.

It’s all about DBus

When I started this library I hadn’t quite appreciated how much it was going to be about DBus. The information at the bottom this link https://www.freedesktop.org/wiki/Software/DBusBindings/ says that python-dbus shouldn’t be used for new applications. Of the list it suggests, it appears pydbus is the most appropriate for this project. This issue https://github.com/LEW21/pydbus/issues/20 did seem like it was going to cause an issue. However it appears that Debian Stretch has a new enough version of GLib that it is OK. Using pydbus removes the need for a lot of code that has been created in the Bluezero library such as adapter.py, device.py and GATT.py

Testing

Testing and coverage metrics have never really been enabled on this library. There are a number of reasons behind this mainly based around restrictions of installing libraries on Travis and the usability of dbusmock library. Not having a usable test strategy for this library is a big (unresolved) issue!

Going forward

Get ready for the release of Debian Stretch:

• Switch from using python-dbus to pydbus.
• Focus on BlueZ 5.43

This leaves the big issue as testing and probably the area where I would most appreciate people's input. The ideal would to get something working that would allow testing on Travis. I believe because of python3-gi package then that limits us to running with Python 3.2 on Travis. Python 3.2 is a problem with the coverage utilities. One possible solution is to use a container system like Docker on Travis. Let’s assume we can fix the python3-gi/Travis issue with containers, then that leaves the issue of how to mock DBus calls as there will be no real Bluetooth hardware. Should effort be put into dbusmock https://github.com/martinpitt/python-dbusmock or look at mocking this a different way?

The other option is to turn away from Travis and mocking and set up actual hardware for automating testing locally that uses real Bluetooth links. This could be Linux to Linux or even Linux to micro:bit with scripts that gets feedback over the network or USB from the remote device. The downside with this is that it is a barrier to other people contributing.

[WayneKeenan]

"The downside with this is that it is a barrier to other people contributing."

Depends, some test hardware could be a shared resource (without direct access to developers) hosted in someones cupboard if Travis has anything like Hudson/Jenkins remote (reverse SSH) agents?

Just need a Pi (or two for periph & central) and a maybe a couple of BLE MCU devices for 'real world' tests.

The reverse SSH, if supported, is handy as it eliminates punching thru firewalls (apart from port 22 outbound), that is, if Travis has such things.

[Godley]

Have you looked into Mythic Beasts RPI hosting? Could be good for this and I'm sure given their rep they'd be happy to help with custom setups. Phil at Pimoroni is also having similar issues, when I spoke to Paul last they were planning to try to get a raspberry pi CI server set up in house so if he's got anywhere with it he might be able to offer some tips. What exactly is python-gi and why is it necessary?

[ukBaz]

Interesting idea about using Mythic Beasts RPi hosting.

python-gi gives you the main loop to process asyncronous Bluetooth events

[ukBaz]

I've been giving this some more thought and I think that having actual hardware might not work.

For example, one of the main pieces of this library is turning known Bluetooth adapter addresses and GATT characteristics into DBus paths. Getting the DBus paths will be done from the GetManagedObjects functionality. Something like:

from pydbus import SystemBus
dbus = SystemBus()
mngr = dbus.get('org.bluez', '/')
mngr.GetManagedObjects()

GetManagedObjects returns something like:

{'/org/bluez/hci0/dev_D4_AE_95_4C_3E_A4':
    {'org.bluez.Device1': {'Appearance': 512, 'Name': 'BBC micro:bit [zezet]', 'Trusted': False, 'Connected': False, 'Adapter': '/org/bluez/hci0', 'Address': 'D4:AE:95:4C:3E:A4', 'Paired': False, 'Blocked': False, 'Alias': 'BBC micro:bit [zezet]', 'ServicesResolved': False, 'LegacyPairing': False, 'UUIDs': ['00001800-0000-1000-8000-00805f9b34fb', '00001801-0000-1000-8000-00805f9b34fb', '0000180a-0000-1000-8000-00805f9b34fb', 'e95d0753-251d-470a-a062-fa1922dfa9a8', 'e95d127b-251d-470a-a062-fa1922dfa9a8', 'e95d6100-251d-470a-a062-fa1922dfa9a8', 'e95d9882-251d-470a-a062-fa1922dfa9a8', 'e95dd91d-251d-470a-a062-fa1922dfa9a8', 'e95df2d8-251d-470a-a062-fa1922dfa9a8']},
     'org.freedesktop.DBus.Properties': {},
     'org.freedesktop.DBus.Introspectable': {}
    },
'/org/bluez/hci0': 
    {'org.bluez.LEAdvertisingManager1': {}, 
     'org.bluez.Adapter1': {'Name': 'RPi3', 'Class': 0, 'Powered': True, 'PairableTimeout': 0, 'DiscoverableTimeout': 180, 'Address': 'B8:27:EB:22:57:E0', 'Alias': 'BluezeroLight', 'Discoverable': False, 'Pairable': True, 'Discovering': False, 'Modalias': 'usb:v1D6Bp0246d052C', 'UUIDs': ['00001801-0000-1000-8000-00805f9b34fb', '0000110e-0000-1000-8000-00805f9b34fb', '00001200-0000-1000-8000-00805f9b34fb', '00001800-0000-1000-8000-00805f9b34fb', '0000110c-0000-1000-8000-00805f9b34fb']}
     }
}

There is not guarantee that BlueZ will give these items the same DBus path names so Bluezero will need a function (e.g. get_dbus_path) that will return the dbus path for given Bluetooth device and characteristics. If I write tests something like:

import unittest
import bluezero

class DeviceAddressTest(unittest.TestCase):
    def test_dev1(self):
        self.assertEqual(bluezero.get_dbus_path('E4:43:33:7E:54:1C'),
                                                '/org/bluez/hci0/dev_E4_43_33_7E_54_1C')

    def test_dev2(self):
        self.assertEqual(bluezero.get_dbus_path('e4:43:33:7e:54:1c'),
                                            '/org/bluez/hci0/dev_E4_43_33_7E_54_1C')

    def test_service(self):
        self.assertEqual(bluezero.get_dbus_path('F7:17:E4:09:C0:C6',
                                                'e95df2d8-251d-470a-a062-fa1922dfa9a8'),
                         '/org/bluez/hci0/dev_F7_17_E4_09_C0_C6/service0031')

    def test_characteristic(self):
        self.assertEqual(bluezero.get_dbus_path('FD:6B:11:CD:4A:9B',
                                                'e95d127b-251d-470a-a062-fa1922dfa9a8',
                                                'e95d5899-251d-470a-a062-fa1922dfa9a8'),
                         '/org/bluez/hci0/dev_FD_6B_11_CD_4A_9B/service0020/char0021')

    def test_descriptor(self):
        self.assertEqual(bluezero.get_dbus_path('EB:F6:95:27:84:A0',
                                                'e95d0753-251d-470a-a062-fa1922dfa9a8',
                                                'e95dca4b-251d-470a-a062-fa1922dfa9a8',
                                                '00002902-0000-1000-8000-00805f9b34fb'),
                         '/org/bluez/hci0/dev_EB_F6_95_27_84_A0/service0013/char0014/desc0016')

    def test_adapter(self):
        self.assertEqual(bluezero.get_dbus_path(adapter='00:00:00:00:5A:AD'),
                         '/org/bluez/hci0')

if __name__ == '__main__':
    unittest.main()

Doing this with real hardware is likely to be an unreliable test because the path names will be different on different runs.

[ukBaz]

The kind of scripts that will need to be tested are like the following that writes 'Hello world!' to the micro:bit LED screen. The paths could be hardcoded into the test with mocks but would need to be replaced with bluezero.get_dbus_path if using real hardware.

from pydbus import SystemBus

dbus = SystemBus()

ubit = dbus.get('org.bluez', '/org/bluez/hci0/dev_D4_AE_95_4C_3E_A4')
ubit.Connect()

text = dbus.get('org.bluez', '/org/bluez/hci0/dev_D4_AE_95_4C_3E_A4/service002a/char002d')
text.WriteValue([72, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100, 33], {})

ubit.Disconnect()

If dbusmock is used then the test needs to check that the correct format is sent to the DBus method. If real hardware is used then I would need to work out how to test the correct thing has arrived.

[ukBaz]

Another example of the kinds of tests that would need to be done is the main loop to process asyncronous Bluetooth events. For example this gets the temperature of the micro:bit

from gi.repository import GLib
from pydbus import SystemBus

dbus = SystemBus()
loop = GLib.MainLoop()

ubit = dbus.get('org.bluez', '/org/bluez/hci0/dev_D4_AE_95_4C_3E_A4')
ubit.Connect()

while not ubit.ServicesResolved:
    pass

temperature = dbus.get('org.bluez', '/org/bluez/hci0/dev_D4_AE_95_4C_3E_A4/service003a/char003b')
temperature.StartNotify()
temperature.onPropertiesChanged = print

def end_notify():
    print('Ending notifications')
    temperature.StopNotify()
    ubit.Disconnect()
    loop.quit()

GLib.timeout_add_seconds(10, end_notify)
loop.run()

Things like temperature and button presses on the remote Bluetooth device are going to be difficult to do integration tests using real hardware.

[Godley]

Would anything about the path names be the same? I'm guessing unlikely. Instead of testing that they're identical, in that case, you could just test that there's one and only one raspberry pi, and one and only one microbit showing in the dictionary of paths. I tend to do this in circumstances where I'm testing a list contains all the elements because it's usually unlikely I'll get the exact same order back from the method every time.

[ukBaz]

Idea

I've been focusing on the mock piece of the problems laid out up above and I have something that is able to test the "hello_world" script. The underlying idea behind the solution is to use mock.patch to replace the pydbus.SystemBus() call in scripts being tested so that they use a mock BlueZ API on the SessionBus rather than the real BlueZ interface on the SystemBus. The key line for doing this is:

mock_bus.SystemBus.return_value = pydbus.SessionBus()

There are a few more details that need to be worked out/polished up but I think this is the most promising of the ideas I've experimented with. It gets around trying to eavesdrop on the real org.bluez on the SystemBus and the various permissions that causes. It also feels cleaner than mockdbus. Creating the mock BlueZ API might be a bit of work unless it is done well so will need some thinking about. pydbus does make this simpler than when we were using python-dbus although pydbus does restrict us to Debian Stretch (or updating GLib on Jessie).

I'll probably leave this issue open for a couple more days to see if anyone has any feedback and I'll continue testings. If all goes well I will close this issue and open two others. One to track implenenting this mock implementation and the other to get a Docker image for Stretch working with TravisCI.

The Details

For reference these are files I've used for testing this idea

The script being tested

import pydbus

def hello_world():
    dbus = pydbus.SystemBus()

    ubit = dbus.get('org.bluez', '/org/bluez/hci0/dev_11_22_33_44_55_66')
    ubit.Connect()

    while not ubit.ServicesResolved:
        pass

    text = dbus.get('org.bluez', '/org/bluez/hci0/dev_11_22_33_44_55_66/service002a/char002d')
    text.WriteValue([72, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100, 33], {})

    ubit.Disconnect()

if __name__ == '__main__':
    hello_world()

The mock BlueZ DBus

At the moment this needs to be started before the tests in a separate window. Need to make this part of the test scripts (probably). Command python3 mock_ubit/mock_ubit.py

from pydbus import SessionBus
from pydbus.generic import signal
from gi.repository import GLib

loop = GLib.MainLoop()

dbus = SessionBus()

class MockDevice(object):
    """
      <node>
        <interface name='org.bluez.Device1'>
          <method name='Connect'>
          </method>
          <method name='Disconnect'>
          </method>
          <property name="Name" type="s" access="read">
            <annotation name="org.feedesktop.DBus.EmitsChangedSignal" value="micro:bit"/>
          </property>
          <property name="Connected" type="b" access="read">
            <annotation name="org.feedesktop.DBus.EmitsChangedSignal" value="true"/>
          </property>
          <property name="ServicesResolved" type="b" access="read">
            <annotation name="org.feedesktop.DBus.EmitsChangedSignal" value="true"/>
          </property>
        </interface>
      </node>
    """
    def __init__(self, mac_address='11:22:33:44:55:66'):
        self.server_address = mac_address
        self.Name = 'ble device'
        self._Connected = False
        self._ServiecesResolved = False

    def Connect(self):
        self._Connected = True
        self._ServiecesResolved = True

    def Disconnect(self):
        self._Connected = False
        self._ServiecesResolved = False

    @property
    def Name(self):
        return self._Name

    @Name.setter
    def Name(self, value):
        self._Name = value
        self.PropertiesChanged('org.bluez.Device1', {'Name': self._Name}, [])

    @property
    def Connected(self):
        return self._Connected

    @Connected.setter
    def Connected(self, value):
        self._Connected = value
        self.PropertiesChanged('org.bluez.Adapter1', {'Connected': self._Connected}, [])

    @property
    def ServicesResolved(self):
        return self._Connected

    @Connected.setter
    def ServicesResolved(self, value):
        self._Connected = value
        self.PropertiesChanged('org.bluez.Adapter1', {'Connected': self._Connected}, [])

    PropertiesChanged = signal()

class MockGatt(object):
    """
      <node>
        <interface name='org.bluez.GattService1'>
          <method name='WriteValue'>
            <arg type='aq' name='value' direction='in'/>
            <arg type='aq' name='options' direction='in'/>
          </method>
          <method name='Disconnect'>
          </method>
          <property name="UUID" type="s" access="read">
            <annotation name="org.feedesktop.DBus.EmitsChangedSignal" value="micro:bit"/>
          </property>
        </interface>
      </node>
    """
    def __init__(self, mac_address='11:22:33:44:55:66'):
        self._UUID = 'E95D93EE-251D-470A-A062-FA1922DFA9A8'

    def WriteValue(self, value, flags):
        pass

    @property
    def UUID(self):
        return self._UUID

    PropertiesChanged = signal()

dbus.publish('org.bluez',
             ('/org/bluez/hci0/dev_11_22_33_44_55_66',
              MockDevice()),
             ('/org/bluez/hci0/dev_11_22_33_44_55_66/service002a/char002d',
              MockGatt()))
loop.run()

The test

from bluezero import hello_world

import sys
import pydbus
import unittest
from unittest import mock

class WriteHelloWorld(unittest.TestCase):
    @mock.patch('bluezero.hello_world.pydbus')
    def test_hw(self, mock_bus):
        mock_bus.SystemBus.return_value = pydbus.SessionBus()
        hello_world.hello_world()

if __name__ == '__main__':
    # avoid writing to stderr
    unittest.main(testRunner=unittest.TextTestRunner(stream=sys.stdout,
                                                     verbosity=2))

This was run as follows:

$ python3 -m unittest -v tests.test_hello_world
test_hw (tests.test_hello_world.WriteHelloWorld) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.153s

OK

[LJBD]

Hello!

I'm not sure if this is the right place to ask, but as you are pondering about the future of the project, I wanted to ask a question. What is the relation of this project to other Python implementations of BLE (e.g. pygatt and gatt-python)? Do you have different attitudes or goals? Have you evaluated them and decided they lack some functionalities?

Best regards, Łukasz

[ukBaz]

Hi Łukasz,

This is as good a place as anywhere to ask the question... :-)

My interest in Bluetooth, and more specifically in BLE, stretches back a few years and was mainly focused on helping a school with a STEM project. This then morphed into helping a couple of people with the Lichen Beacon project for which I had to do with Node JS beacuse of a lack of Python libraries. I wanted to still help schools work with BLE (especially as the micro:bit got launched) using Python and so Bluezero was created. The idea was to fit in with the other *Zero projects. Before I started this project the other Python libraries were not really focused on BLE. The libraries that did (like the ones you've linked to) often used the BlueZ command line tools under a python wrapper. One of the challenges with this approach is that there is no fixed API for these command line tools and some of the command line tools are being deprecated.

As you can see the Bluezero repo has hit a number of problems and as this is a "hobby" project we just don't have the resources to work around those issues. We have to take the view that we will wait for those issues in the dependancies to get fixed. I fully expect to get overtaken by a better resourced project (it just doesn't seem to have happened yet). I was optimistic when I saw the Adafruit Python BluefruitLE library get started however it doesn't seem to be very actively developed currently. It does seem to have thrown up that there might be a fundamental performance problem with the BlueZ DBus approach.

This is turning out to be a longer response than I imagined... so to try to wrap this up, from my research these doesn't seem to be a clear winner (Bluezero included) when it comes to Python BLE libraries. Although I am always happy to be proved wrong :-)

Cheers, Barry P.S. I feel it would be wrong not to mention a couple of other repositories to look at: IanHarvey / bluepy karulis/pybluez

[WayneKeenan]

I hadn't seen a couple of those and haven't peeked under the bonnet of most so I thought I would have a quick look around and summarise:

bluezero Central & Peripheral (DBus) bluepy Central & Peripheral (Custom binary helper) pygattlib Central Only (Python C Module, using HCI API) gatt-python Central Only (DBus) pygatt Central Only ('gatttool' wrapper) pybluez Central Only (BLE provided by 'pygattlib') [EXPERIMENTAL] adafruit Central Only (DBus) [INACTIVE]

DBus seems to introduce a lot of 'opportunities'... (yes, read that as sucks all the life blood out of development fun)

Pygattlib, although Central only, is the only one (on the list, there may be others) that uses the standard Bluetooth HCI API and has implemented it as a Python module, not an external native library, which on the surface seems like a very good idea to me; potential performance benefits over DBus, no messy child processes and HCI is cross platform and not tied to an OS Messaging framework.

Perhaps one way forward is to swap DBUS for HCI, non-trivial and a huge uphill start I know.

Perhaps another way forward is to turn BlueZero into a pure-python 'zero' wrapper around Pygattlib and add/wait for Peripheral support to be added to Pygattlib. But where do you draw the line(s) between reuse, ease of setup and too many moving parts...

[LJBD]

@ukBaz, @WayneKeenan - thanks a lot for your the comprehensive answers!

Reason for writing this

Since my question has spurred you to write them, I feel entitled to provide my opinion on the matter as well. Please, bear in mind that it's coming from someone who has just started his BLE adventures last week, though.

What I think of the existing libraries

I believe that a good library shouldn't be based on a soon-to-be deprecated parts of bluez. If, as you say, DBus is such a pain in the ass, then it should also avoid that. And that leaves us with pygattlib and bluepy.

The thing that drove me away from bluepy is its code style. It looks like being written by someone coming from another language and just starting coding with Python. I've found it difficult to follow.

I've looked at pygattlib and it seems nice - there's only one drawback to it: it uses boost as the Python extension framework. From my experience (which isn't big when it comes to Python C++ extensions), Boost is a thing of the past. It also tends to be hard to maintain if your project grows bigger. I'm somewhat involved in pytango which is based on Boost and the general consensus amongst the maintainers is that the only thing keeping us from moving away from Boost is the lack of manpower and/or time to do so.

Right now, I'd use Cython, since it seems that it's becoming the most popular extension framework. It may be a good idea to do the transition now because pygattlib looks small enough before the peripheral role is added. Which to me is a must if the library is to become the best Python solution for BLE. I also like the notion of bluezero becoming a pure-Python part of it. Maybe the two projects could be merged together?

Other things on my mind

1. There are also other criteria by which the libraries should be assessed. Just to name a few:
   • being cross-platform,
   • supporting Python 3.x (the higher x, the better),
   • having automated tests,
   • having good documentation.
2. AFAIK, Bluetooth is asynchronous. In bluezero, you use GObject.mainloop to handle events. Maybe it would be worth to try to switch to a more "pythonic" solution - asyncio? It's Python 3 - only, though, but there's a deprecated port of it to Python 2.
3. BlueZero looks great in terms of the code style - kudos to you, guys.
4. Using Docker for Travis testing is a great thing. The other projects we're using at the synchrotron facility where I work - Taurus and Sardana - use it and they're really happy with it. I guess that it wouldn't be too complicated to create your own Docker testing setup if there's a good way of mocking DBus. This issue might also have some relevant information.

Woah, there was a lot of thoughts swirling around my head the past several days...

[ukBaz]

Thanks @WayneKeenan and @LJBD. This is a very helpful discussion for me. There are a couple of strands I wanted to pick out for some extra comment

... DBus...

The advantage of the DBus API is that it is the high level API provided by BlueZ and removes the need for developers to have low level knowledge of the Bluetooth specification. The downside of this API is that there is a risk of flooding the DBus software bus. This appears to be the cause of the performance issue I linked to. It is also why the BlueZ developers will not implement BLE Beacons in the DBus API.

BlueZero into a pure-python 'zero' wrapper around Pygattlib

I'll take a more detailed look at Pygattlib to see what that would actually mean

There are also other criteria by which the libraries should be assessed. Just to name a few: being cross-platform,

Agreed. Although Windows is behind the curve with Bluetooth and BLE in particular. Bluetooth is very closely linked to hardware. I've been looking at the Web Bluetooth efforts to see what is really possible. If that better resourced project is struggling to go cross-platform for Bluetooth then I have to keep my goals focused on Linux for now.

supporting Python 3.x (the higher x, the better),

I have always wanted to support as broad a range of Python versions as possible.

having automated tests,

Again, agree with this. Web Bluetooth seem to have done a good job of separating and mocking the hardware. Until there is something similar for Python then this will continue to be an issue. If we go to the HCI API (rather than DBus) this will be even harder I suspect.

having good documentation.

Agreed

AFAIK, Bluetooth is asynchronous. In bluezero, you use GObject.mainloop to handle events. Maybe it would be worth to try to switch to a more "pythonic" solution - asyncio? It's Python 3 - only, though, but there's a deprecated port of it to Python 2.

I have looked at trying to use asyncio. It wasn't obvious to me how to use that with DBus especially when the only main loop supported by dbus is GLib.

BlueZero looks great in terms of the code style - kudos to you, guys.

Thanks for the nice feedback although much of the kudos must go to @mr499 for this. > Using Docker for Travis testing is a great thing. Agreed. Although abstracting away the hardware is the key to being able to develop a proper test strategy. > Woah, there was a lot of thoughts swirling around my head the past several days... Thank you for sharing. It has been good. # And finally... Now that I have an early version of Debian Stretch for the Raspberry Pi I'm going to look at creating a workshop for communicating with a micro:bit. This might kick start my activity on this library again. Ultimately it comes down to resources and currently there isn't a well resourced project needing a generalised Python Bluetooth API. :-(

[WayneKeenan]

The advantage of the DBus API is that it is the high level API provided by BlueZ and removes the need for developers to have low level knowledge of the Bluetooth specification.

Aren't there 2 categories of developers: end users of the bluezero library and maintainers of the bluezero library? Where the former group shouldn't really care whats inside bluezero and perhaps one goal of the later groups' is to hide the details to make bluezero the high level Python BluetoothLE API (?)

Now that I have an early version of Debian Stretch for the Raspberry Pi I'm going to look at creating a workshop for communicating with a micro:bit. This might kick start my activity on this library again.

ooo, is it worth trying out yet? where?

[ukBaz]

Hi @WayneKeenan ,

Aren't there 2 categories of developers: end users of the bluezero library and maintainers of the bluezero library? Where the former group shouldn't really care whats inside bluezero and perhaps one goal of the later groups' is to hide the details to make bluezero the high level Python BluetoothLE API (?)

I agree with you. The end users shouldn't/doesn't care what's inside. And while BlueZ think of the DBus API as high level, many Pythonistas while not. or at least not very Pythonic. Bluezero's goals can be met using either the DBus or HCI interfaces. However, I think the workload for development is quite different. HCI looks rather daunting from where I am at the moment...

However, I'm always open to being persuaded.

Does anyone have a suggestion on how to implement the raw HCI socket example in Python? https://git.kernel.org/pub/scm/bluetooth/bluez.git/tree/doc/mgmt-api.txt#n90

If we can do that then maybe we test it by reading the status of the controller: https://git.kernel.org/pub/scm/bluetooth/bluez.git/tree/doc/mgmt-api.txt#n253

And issues a command to turn the Bluetooth on/off https://git.kernel.org/pub/scm/bluetooth/bluez.git/tree/doc/mgmt-api.txt#n318

Finally, if we can then read a beacon I'll be convinced this is a plausible route that has benefits over DBus :-)

[colin-guyon]

Hi, about usage of HCI commands, I started some time ago a small python package for my own needs: https://github.com/colin-guyon/py-bluetooth-utils

It either uses HCI commands using PyBluez, or does ioctl calls like it's done in Bluez tools such as hciconfig. Main functions:

• toggle_device : enable or disable a bluetooth device
• set_scan : set scan type on a device ("noscan", "iscan", "pscan", "piscan")
• enable/disable_le_scan : enable BLE scanning
• parse_le_advertising_events : parse and read BLE advertisements packets
• start/stop_le_advertising : advertise custom data using BLE

It does not contain lots of things, but just in case it can help...

[WayneKeenan]

Thanks @colin-guyon , that's very interesting, especially as it's pure Python and also for the link to the Beacon scanning example, I think that might be a great help for convincing @ukBaz :)

[WayneKeenan]

Also, I understand that by using the HCI API like this it means the user land portion of bluez is no longer required, as the socket based HCI API is provided by the bluez kernel driver.

So no need to write against this weeks flavour of the BlueZ DBUS API or make a user build the latest bluez bluetoothd from source. Maybe just a case of stopping the existing bluetoothd service (if that) ?

@ukBaz are you convinced yet? :)

[ukBaz]

The only thing that I feel uneasy about is these use:

# import PyBluez
import bluetooth._bluetooth as bluez

I keep looking at https://docs.python.org/3/library/socket.html?highlight=bluetooth#socket-families and thinking that BTPROTO_HCI should be able to help us.

[LJBD]

AFAIU @colin-guyon's py-bluetooth-utils use PyBluez which in turn uses PyGATTLib for BLE support - am I right? This means that the foundations for the pure Python wrapper around the latter are already done, doesn't it? Or am I missing something here?

[colin-guyon]

@LJBD py-bluetooth-utils uses some constants from PyBluez and also few low level functions, like hci_send_cmd that seems to be calling the function of the same name of the bluez C API. So in py-bluetooth-utils PyGATTLib is not used. But indeed for BLE related APIs of PyBluez it seems PyGATTLib is used (but I don't know this lib)

[WayneKeenan]

@LJBD Since my previous post I now know that anything that directly or indirectly pulls in the bluetooth._bluetoothlibrary is using the BlueZ C compiled library, which I think is most, if not all of the non D-Bus based Python bluetooth libraries, AFAIK. Using the Socket based HCI approach really is pure python and doesn't require the BlueZ python/C or Bluetoothd service at all. This I'm now 100% sure about... :)

[sandeepmistry]

Apologies, I haven't had time to read the whole thread yet, but here are some lessons learned from the noble/bleno Node.js world:

1. There are some requests to use the D-Bus API: https://github.com/sandeepmistry/noble/issues/310
2. Unless you use user channel HCI sockets (which requires the adapter to be powered off on app start), the kernel can interfere. This causes issues with connection intervals or connections dropping.
3. bleno cannot work with BlueZ bluetoothd running, as it has a built-in GATT server running.

Hope this helps, let me know if you have any more specific questions :)

[ukBaz]

Hi,

Thanks to everyone that has contributed to this thread. It has been a very useful discussion.

This thread has become very long and unwieldy so I am going to close it. There are a number of other issues opened for the various topics arising from this discussion so please use them or open a new issue.

Feedback to BlueZ project

As was correctly mentioned on Twitter, there is some feedback in this thread that should shared with the BlueZ project as it may be helpful for them. This has been done through the BlueZ mailing list. Details of that feedback is at: http://marc.info/?l=linux-bluetooth&m=150238485227628&w=2

Investigate using Python HCI rather than DBus API.

Wayne has done some work to this end with a proof of concept at: https://github.com/TheBubbleworks/python-hcipy

Discussion on HCI can be covered on #133 if people want to ask questions or anything. There is also #137 to stop DBus calls leaking in to lots of files. This should make moving to HCI (or other OSes) easier.

Testing

With #135 I've moved this library to using Python unittest.mock rather than dbusmock. This is really for ease of getting things running on Travis. While far from ideal it does improve things. It would be much appreciated if people could take a look at writing a test. Please look for the low hanging fruit so coverage can be improved on the repository. If people have questions then please open an issue with a question. There are lots of opportunities for improvement: https://codecov.io/gh/ukBaz/python-bluezero/tree/master/bluezero