[Question] Micropython bluetooth module in Pybricks?

[gyenesvi]

Question I have been browsing the Micropython documentation, which mentions a bluetooth module with BLE api. However, if I try to import bluetooth, as the Micropython examples suggest, the module is not found. Is that module not available in the Pybricks distribution of Micropython? If not, what is the reason it is not exposed? Firmware size?

Context I would like to experiment with Pybricks to build bluetooth connection to BLE gamepad controllers to see if I can read their input and control motors with them. The Micropython bluetooth libraries contain example code to communicate with HID devices, so it looks like it should be doable if the bluetooth module would work on Pybricks.

[laurensvalk]

Welcome @gyenesvi --- good to see you here!

The big difference is that most MicroPython boards are programmed via USB. This means you can get very creative with Bluetooth/BLE and nothing will ever go wrong.

By contrast, we use BLE for programming the LEGO hubs. Any custom BLE behavior with the bluetooth module would get in the way of this unfortunately.

So in the long term we might need dedicated solutions for things like HID interfaces. See #262 for more details.

[gyenesvi]

Thanks @laurensvalk for the quick response! Oh, I didn't think about that, I see. BTW, what would you consider such custom behavior that is potentially dangerous? Can you think of an example? Does connecting to other devices have such dangers already?

Yes, I know about #262 I have read through it, and I'd like to help in speeding up 4. becoming a reality. I did also look into the C code of Pybricks, found the bluetooth handling parts for the lego remote. Unfortunately, it seems now that it even there, the BLE connectivity code is tied to the lego remote, there don't seem to be generic (~mid level) APIs. Any chance of cleaning that up a bit so that it would become more reusable?

So if the idea then is to expose only a HID specific API to python, I wonder how much work that would be? I am guessing that the HID related stuff is already present on the C side of micropython as well, though I did not find that yet. Do you know about that? Furthermore, do you have any guesses of how involved it would get to make a HID API? I mean does it look like exposing existing C functionality through a Python interface, or is it more like coding much of it on lower levels on the C side first?

[laurensvalk]

It's not dangerous, just not a good user experience if you have to reset your hub just to try again.

Unfortunately, it seems now that it even there, the BLE connectivity code is tied to the lego remote, there don't seem to be generic (~mid level) APIs. Any chance of cleaning that up a bit so that it would become more reusable?

That's precisely it. Once cleaned up, adding new stuff should be easy :smile:

So I think the issue isn't necessarily a technical obstacle, but rather about finding the right prioritization of feature requests and time to get it done. Right now, we are focused on bringing the current beta to the stable channel, for example.

[dlech]

The main reason we don't have any generic Bluetooth code is that the Bluetooth chips on the smaller hubs are quite limited. So we probably aren't likely to add much to them. But the SPIKE/MINDSTORMS hubs use BTStack and have much more RAM and flash, so we could consider adding more there.

So if you are interested in looking into it farther, looking to see if BTStack has some HID client code we could use would be a good start.

For a Python API, we would want something that could be used in a non-blocking way, so some sort of event queue that could be polled.

[gyenesvi]

So I think the issue isn't necessarily a technical obstacle, but rather about finding the right prioritization of feature requests and time to get it done.

Sure, I understand that. What I was wondering is how I could get involved easily, because that would mean no extra time required on your side. If I could work in parallel, maybe that would make it worth prioritizing some cleanup to provide me with some basis?

BTW, is there an expected timeline for publishing the current beta version?

@dlech thanks for chiming in!

Bluetooth chips on the smaller hubs are quite limited.

I'd definitely be interested in the 4-port Technic hub for technic builds, where does that fall? The Mindstorms/Spike hubs can also be useful for larger builds, but less technic people have them. The 2-port City hub may also be interesting for technic people, is that too small already?

BTStack definitely seems to have HID profile api:

If I understand correctly, the lego hub would be a host in this scenario (gamepad would be the device), for which the relevant apis docs are here: https://bluekitchen-gmbh.com/btstack/#appendix/hid_host/

And this example seems relevant to me: https://github.com/bluekitchen/btstack/blob/master/example/hog_host_demo.c

However, it is still not clear to me how HID is positioned in the BLE stack. For example, HID Over Gatt (which would be needed here for BLE if I am right) suggests to me that HID is built on top of the Gatt services, so I was guessing that all the connection handling is done by regular BLE stuff, that I have some experience with (such as scanning and connecting devices), and then the HID stuff would be like on top of that, decoding the HID protocol messages. But it seems from some examples that the HID API has it's own ways of scanning / connecting to devices, so not sure if it actually uses those BLE apis underneath (wraps them), or is it somewhat different? Any ideas about that?

Still trying to estimate the depth of this task to see at what level I could join in..

[dlech]

The Technic and City hubs both have the same TI CC2640R2 chip. For those, the Bluetooth stack runs mostly on on the Bluetooth chip and we get vendor specific HCI commands/events sent to the MCU. So we can't use BTStack there - any code for these hubs would likely have to be written from scratch.

A first step would be to rework the current pbio/drv/bluetooth code to have queues for both commands and events and rework the connection model to allow more generic connections.

Another thing to consider is that we are already having issues with those chips running out of memory and crashing sometimes, so there is some risk that even if we do the work, it might not be a great end user experience.

Any ideas about that?

Sometimes BTStack uses synthetic events for things that didn't actually come from the Bluetooth chip directly. So I think that might be the case for HID - it could be unpacking the GATT notification and turning it into other HCI "events" specific to the HID protocol.

[gyenesvi]

The Technic and City hubs both have the same TI CC2640R2 chip. For those, the Bluetooth stack runs mostly on on the Bluetooth chip and we get vendor specific HCI commands/events sent to the MCU. So we can't use BTStack there - any code for these hubs would likely have to be written from scratch.

Oh, that's a bummer. It makes it more complicated than I thought. BTW, how does the bluetooth module in Micropython achieve this? It does have an implementation for this chip as well, right? Isn't it possible to reuse generic code from there without exposing the full module?

A first step would be to rework the current pbio/drv/bluetooth code to have queues for both commands and events and rework the connection model to allow more generic connections.

So as far as I see, pbio/drv/bluetooth/bluetooth_stm32_cc2640.c is the relevant implementation here, and it already has most of the basic bluetooth connectivity functionality implemented for the sake of connecting to the remote and other hubs for example. So for basic connectivity, I guess not much need to be added, it would be roughly the same for HID devices as well.

The way I imagine it from what I have read so far, is that HID is just another service profile definition, so upon discovery, the system needs to be able to filter for that service being advertised by the device (and more specifically that it's a gamepad), just as it filters for the lego protocol service uuid in case of connecting to the lego hub/remote. Once connection is done, it seems to me that it's about reading (binary) messages from the relevant characteristic, and decoding the messages (just as in case of lego wireless protocol). This is the more HID specific part that needs to be reimplemented I guess. However, if the implementation would shoot only for gamepads, and not any HID device in general, that part could get simplified as well.

So if the first part about generic bluetooth connectivity would get cleaned up, I could try and give the HID specific part a go. In the meantime I might be able to do some experiments for connecting to a gamepad from my laptop using a generic bluetooth python library and decoding the messages if it is really the way I think it is.

Another thing to consider is that we are already having issues with those chips running out of memory and crashing sometimes, so there is some risk that even if we do the work, it might not be a great end user experience.

Do you mean those TI chips that handle bluetooth? So the memory problems are related to the bluetooth implementation and not for example to the rest of the firmware or the way the end user writes its program on the python side?

[dlech]

BTW, how does the bluetooth module in Micropython achieve this?

MicroPython has two implementations, one based on BTStack and one based on Nimble.

It does have an implementation for this chip as well, right?

No, since this chip uses vendor-specific commands only, it does not work with those.

Isn't it possible to reuse generic code from there without exposing the full module?

Yes, this could probably be done for simple things like reading and writing a characteristic that wouldn't break other uses of BTStack in our code.

I might be able to do some experiments for connecting to a gamepad from my laptop using a generic bluetooth python library and decoding the messages if it is really the way I think it is.

Depending on the OS, this might not be possible (e.g. Windows completely hides HID from generic BLE APIs). However, using a generic BLE library or capturing packets with Wireshark would be useful to see what the advertising data looks like.

Do you mean those TI chips that handle bluetooth? So the memory problems are related to the bluetooth implementation and not for example to the rest of the firmware or the way the end user writes its program on the python side?

The problem is with the firmware that runs on the Bluetooth chip itself. If we try to do too much with it or don't do things in an exactly right way, the chip will just crash and stop responding and it has to be reset.

[dlech]

Also, @kubastastny was working on this a few months ago in #191. Not sure if they made any progress.

[gyenesvi]

Thanks for the info, I get it about Micropython / BTStack not supporting this chip. I was assuming it does have because I thought the original Lego FW is also based on Micropython. Do you know if that is the case? Or is that not public? Even if it is, it can be a modified version I guess.

Yes, this could probably be done for simple things like reading and writing a characteristic that wouldn't break other uses of BTStack in our code.

However, I am not sure what you mean here. What could be reused if Micropython does not have an implementation for this chip? Are you saying that an implementation for such things could be possible without using BTStack, but in a way that it does not interfere with other uses of BTStack? Is BTStack used for something else on the Technic Hub? I was assuming that on each chip, it's either BTStack or something else, so two BT APIs can never interfere.

Anyway, I was wondering that if on the C side, it is possible to make some generic but simplified BT scan/connect and characteristic read/write API, then could the same thing be exposed on the Python side as well? That would make building the HID specific part on it much simpler, and debugging would also be easier.

Depending on the OS, this might not be possible (e.g. Windows completely hides HID from generic BLE APIs). However, using a generic BLE library or capturing packets with Wireshark would be useful to see what the advertising data looks like.

I was hoping that it would not be the case on MacOS, I know iOS does that too.. I'll see.

Also, @kubastastny was working on this a few months ago in https://github.com/pybricks/support/issues/191. Not sure if they made any progress.

Thanks for the reference, it seems they are making some slow progress but maybe got stuck, I'll look into it.

[laurensvalk]

I thought the original Lego FW is also based on Micropython. Do you know if that is the case?

SPIKE Prime, SPIKE Essential, and MINDSTORMS Robot Inventor all have original firmware based on MicroPython. I think they all use BT Stack for Bluetooth as well. All the other hubs (Move Hub, Technic Hub, City Hub) use other proprietary firmware out of the box. We have no idea what the code looks like, but we're pretty sure it's not based on MicroPython. Based on what we know about the Bluetooth chips as David explained above, we also don't think BT Stack is used on these hubs.

Anyway, I was wondering that if on the C side, it is possible to make some generic but simplified BT scan/connect and characteristic read/write API, then could the same thing be exposed on the Python side as well? That would make building the HID specific part on it much simpler, and debugging would also be easier.

This would be a way indeed. Basically, you can expose any C functionality with a (Micro)Python API of your choice.

But as a way of getting started hacking....

So as far as I see, pbio/drv/bluetooth/bluetooth_stm32_cc2640.c is the relevant implementation here, and it already has most of the basic bluetooth connectivity functionality implemented for the sake of connecting to the remote and other hubs for example. So for basic connectivity, I guess not much need to be added, it would be roughly the same for HID devices as well.

... this could be a place to start if you are after some quick results. Instead of refactoring a lot of code to make it more generic, you could start by just replacing the existing Powered Up Remote code by something that talks to your HID device.

This might give you some ideas about what works (and what doesn't), which can serve as inspiration (for you, and us) for the the more thorough rewrite of the Bluetooth code at a later stage.

[dlech]

However, I am not sure what you mean here.

I was only referring to hubs that use BTStack, not the Technic hub.

[kubastastny]

Hi guys (@gyenesvi, @dlech) sorry for the late response. Right now I'm stucked as you mentioned. Because I prioritized work and familly I had a very little progress. I tried to decode captured Vendor specific commands but I could not get the right commands out of the read buffer even though I've seen the communication via Dongle. Then I experienced 2 bugs in Bluez where the connection to the hub (loading of pybrics) was corruped and the error propagated to my local copy via pipx. The first time I experienced this thing cost me 2 nights so once I figured out there is an issue in the library itself I let it be until wheels got the bugfix. Then it worked for some time but the second time it happened hurt my motivation a bit :-( The main reason for me is to play with kids with remote controlled builds so in order to push the idea forward I bought LEGO mindstorms 51515 which supports the same remote controll (xbox BLE). Creating programs is fun with kids so I can include them in the thought process which is great but there are couple problems I'm facing: 1) Using remote control with Minstorms App is considerably lagging 2) There is no way to connect Technic Hub 3) It needs phone or laptop running

I'd like to see the support for the remote control gamepads in pybricks because it solves all issues mentioned above but I don't know whether I'll have a bandwith to do it myself mainly because the painful debugging process. Is there any possibility to debug the connectivity on the Large Hub via USB and reuse the code in case of TechnicHub?

From my experience I've never reached the state to get any notifications from the gamepad on Technic Hub. But I sniffed the data by Samsung phone where I was able to identify and decode 16bit notifications. In case you are interested in it I can provide it to you. These notifications start after the pairing of devices was achieved.

I still have this project on the TODO list but so far it has a low priority.

[dlech]

As discussed, for technical reasons, we won't be able to enabled the upstream MicroPython Bluetooth module, but we are exploring other options to connect to different devices.