The future of BLE, Bluetooth, and USB

[laurensvalk]

Introduction

This issue gathers some ideas for Bluetooth and USB to help us keep track of the big picture while we work on low-level details.

Note that most of this won't be done any time soon (if ever). This is mainly intended to avoid implementation choices now which might come back to hurt us later. If you want us to prioritize this, please consider becoming a sponsor :rocket: .

Nomenclature

• The code in boxes is pseudocode for commands in user scripts.
• Arrows: wireless or wired communication.
• Each color is a different service/characteristic which may or may not share a common connection.

1. Single-hub coding scenario

This is what we are working on today. We are working on a Pybricks BLE service to handle things like starting and stopping programs in a clean way. Standard I/O (print/input) may be a separate service or characteristic on the same connection.

Until now, both were handled on one and the same characteristic, which made it easy to mess up the connection by sending the wrong characters.

Cleaning this up and documenting it also paves the way for other editors or extensions to support Pybricks.

This is the only use case we will target for the upcoming v3.0 release. That is likely also the last release for Move Hub; it won't have enough space for the other features listed below.

2. Multi-hub coding scenario

This might work as above, but there are many elements still missing such as:

• Distinguishing hubs (e.g. by name or color)
• Multiple Pybricks Code instances (e.g. tabs or windows)

3a. Multi-hub communication scenario

This is still to be explored. At the basic level, there might be serial i/o streams between hubs that could be used with standard MicroPython tools. Higher level abstractions such as mailboxes could be added on top of these later.

3b. Other BLE UART devices

This is really the same as above, but worth showing in a separate diagram because this really opens up interesting possibilities.

4. BLE HID Peripherals

We may support connecting to generic HID devices over BLE (and possibly classic). Users could build on this to add specific mappings for certain devices like popular gaming consoles.

5. Bluetooth classic scenario

This is only available on Prime Hub and Inventor Hub. Therefore, we will not use Bluetooth classic for any system purposes. So, you won't be able to download and run programs with it.

Rather, it might be used within end-user scripts. For example, you might set up a generic RFCOMM connection with a server or client. This could be another Prime Hub, an EV3 Brick, a laptop, and so on.

6. LEGO Powered Up Remote

See #186.

7. USB scenario

This is only available on Prime Hub and Inventor Hub. Therefore, we will probably not use USB for system purposes. So, you probably won't be able to download and run programs with it. USB will be used primarily for charging the hub. We may re-enable REPL over USB if users want this. Since REPL is treated as a regular program, we'd have to define a way to start and stop it. Perhaps USB can also be used for fast(er) file transfer.

[westurner]

Does there need to be a gamepad Bluetooth HID in the pybricks firmware?

• https://github.com/adafruit/Adafruit_nRF52_Arduino/issues/228#issuecomment-463191651)
  • https://github.com/sparkfun/circuitpython/blob/673d2fb4b7c5e077fc5579af1d249438d17d93ad/tools/hid_report_descriptors.py#L177-L207
  • https://github.com/sparkfun/circuitpython/blob/673d2fb4b7c5e077fc5579af1d249438d17d93ad/tools/hid_report_descriptors.py#L246-L276
  • https://learn.adafruit.com/custom-hid-devices-in-circuitpython?view=all#report-descriptors-3103536
• https://github.com/lemmingDev/ESP32-BLE-Gamepad
• https://github.com/DJm00n/ControllersInfo -- HID profiles for Xbox 360, Xbox One, PS4, PS4, Stadia, and Switch
• "HID over GATT Profile (HOGP) 1.0" https://www.bluetooth.org/docman/handlers/downloaddoc.ashx?doc_id=245141 : 1 Introduction The HID over GATT profile defines the procedures and features to be used by Bluetooth low energy HID Devices using GATT and Bluetooth HID Hosts using GATT. This profile is an adaptation of the USB HID specification [2] to operate over a Bluetooth low energy wireless link. This profile shall operate over an LE transport only. For BR/EDR, the Bluetooth Human Interface Device Profile Specification [9] shall be used. 1.1 Profile Dependencies This profile requires the Generic Attribute Profile (GATT), the Battery Service, the Device Information Service, and the Scan Parameters Profile. This specification can be used with Bluetooth Core Specification Version 4.0 [3] or later.

[laurensvalk]

Thanks everyone! Please move the gamepad findings and discussion over to https://github.com/pybricks/support/issues/1024. Thanks!

[mhornoloopers]

I am interested in the possibility of using a USB connection to run Python code on the Spike Prime Hub. I understand that this may not be a readily available feature, but I would like to explore any potential solutions or workarounds.

If there is no existing method to achieve this, could you please provide some guidance or suggestions on how I might approach implementing a USB connection for this purpose? Any information or resources you can share would be greatly appreciated.

[laurensvalk]

Do you have a particular reason why Bluetooth can't be used?

[mhornoloopers]

Do you have a particular reason why Bluetooth can't be used?

My client's users are already used to using USB to connect their lego robots and having them change to Bluetooth is not an option unfortunately.

[laurensvalk]

Thanks for the response. I'm still not sure that I follow - no user action is required to use Bluetooth in pybricks (no need to press the Bluetooth button). You can still have USB plugged in for charging.

In other words, whether it not the cable is plugged in, the user work flow is the same.

Can you elaborate on why this won't work? Just trying to get a better picture to see if this would help other users, too.

[mhornoloopers]

The type of user are kids so it's more difficult for them to stablish a bluetooth connection rather than just plug an usb (and the teachers are also used to the usb connection).

no user action is required to use Bluetooth in pybricks (no need to press the Bluetooth button)

I'm highly curious about this. How would this work? Should there not be some prior configuration for the connection to be established between the device and the computer?

Even so there's also another two reasons and those are that:

• In a classroom where several Spike Prime Hubs are used at the same time, so they tend to connect to the wrong hubs.
• Some computers don't have Bluetooth (desktop computers)

[westurner]

• https://www.raspberrypi.com/news/raspberry-pi-build-hat-lego-education/
  • https://www.raspberrypi.com/products/build-hat/

    Raspberry Pi meets LEGO® Education The Build HAT (Hardware Attached on Top), priced at $25, is a new add-on board for your Raspberry Pi. It connects to the 40-pin GPIO header and can be used to control up to four LEGO® Technic™ motors and sensors from the LEGO® Education SPIKE™ Portfolio.

    The new HAT works with all 40-pin GPIO Raspberry Pi boards, including Raspberry Pi 4 and Raspberry Pi Zero. With the addition of a ribbon cable or other extension device, you can also use it with Raspberry Pi 400.

Raspberry Pi (and other compatible SBCs with a 40 pin GPIO) have USB and Power.

USB-C and/or USB-A on a new or refreshed LEGO hub product may also solve for your use cases

On Wed, Apr 5, 2023, 7:59 AM Marcos Horno @.***> wrote:

The main reason is user action. The type of user are kids so it's more difficult for them to stablish a bluetooth connection rather than just plug an usb and them and the teachers are already used to the usb connection.

• no user action is required to use Bluetooth in pybricks (no need to press the Bluetooth button) I'm highly curious about this. How would this work? Should there not be some prior configuration for the connection to be established between the device and the computer?

— Reply to this email directly, view it on GitHub https://github.com/pybricks/support/issues/262#issuecomment-1497368312, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAMNS4UD2ZPHYMKFXFKLMDW7VNCRANCNFSM4YIEEWRQ . You are receiving this because you were mentioned.Message ID: @.***>

[laurensvalk]

I'm highly curious about this. How would this work? Should there not be some prior configuration for the connection to be established between the device and the computer?

You click the connect button in the app. Give it a try if you like. Since this is a web app, the procedure would be the same for USB. The browser won't let us auto-connect for security reasons.

You could also use the official LEGO apps with Python if you prefer.

[mhornoloopers]

I'm highly curious about this. How would this work? Should there not be some prior configuration for the connection to be established between the device and the computer?

You click the connect button in the app. Give it a try if you like. Since this is a web app, the procedure would be the same for USB. The browser won't let us auto-connect for security reasons.

You could also use the official LEGO apps with Python if you prefer.

Thank you for the information on Bluetooth connectivity. However, due to our users' specific requirements I mentioned earlier, we are seeking a USB-based solution and are curious if there's a way to implement it with Pybricks.

Is it not possible to achieve this, then? If you have any suggestions or insights, I'd appreciate your input. I understand that this might be a complex task, but any guidance would be valuable.

[laurensvalk]

It's certainly possible technically. We get a lot of feature requests, so we normally try to prioritize features that help the most users. We do also work with commercial partners on some new features, and USB support could be a good example of that --- feel free to reach out at team@pybricks.com if you'd like to learn more.

[lawrencet149]

Another reason for USB support is so we can use pybricks for First Lego League Challenge competitions. Our FLL organization here does not allow bluetooth during the competitions and it is crucial to be able to tweak the code and make adjustments based on previous runs on the tables. I have been teaching my group of kids how to use pybricks and it would be awesome if we can stick with pybricks for the robot competitions this coming season. Thank you for pybricks!

[KumphanartDansiri]

Another reason for USB support is so we can use pybricks for First Lego League Challenge competitions. Our FLL organization here does not allow bluetooth during the competitions and it is crucial to be able to tweak the code and make adjustments based on previous runs on the tables. I have been teaching my group of kids how to use pybricks and it would be awesome if we can stick with pybricks for the robot competitions this coming season. Thank you for pybricks!

Me too. I just get spike prime for my kids to upgrade from ev3 to enter WRO Competition. During competition wireless connection is not allowed. So, either run pybricks in ev3 or move to lego officials for spike prime.

[westurner]

Let's see if I can summarize weeks later from a deep dive on a weekend iirc:

Is this a usable UX workflow for connecting a bluetooth ble gamepad to a LEGO hub?:

• [ ] pybricks/code: Scan for gamepad devices with WebBluetooth, connect, and test that the gamepad(s) work in the browser with WebBluetooth first

  • https://github.com/luser/gamepadtest/blob/master/gamepadtest.js
  • https://github.com/topics/gamepad-api
  • https://github.com/davidcmoffitt/gamepad-controller-test
  • https://github.com/ArunMichaelDsouza/joypad.js
  • https://github.com/alvaromontoro/gamecontroller.js
    • lists the W3C Gamepad API events and button names in the README
  • https://github.com/alvaromontoro/gamepad-simulator

    This module creates a virtual gamepad on the screen that triggers the same events that a real gamepad gamepad would.

• [ ] Transfer the gamepad pairing information (device id,) to the hub along with the micropython src

• [ ] Have the lego hub wait/retry/loop to BLE pair (nonblockingly, in a thread if that's possible w/ micropython?) with the gamepad

• [ ] MicroPython does not implement WebBluetooth, so there probably must be a Gamepad HID (as described here: https://github.com/pybricks/support/issues/262#issuecomment-1493287515 ) in the firmware?:

  • https://github.com/DJm00n/ControllersInfo HID profiles for Xbox 360, Xbox One, PS4, PS5, Stadia, and Switch "HID over GATT Profile (HOGP) 1.0" https://www.bluetooth.org/docman/handlers/downloaddoc.ashx?doc_id=245141

• [ ] MicroPython: wrap the HID over GATT Profile events and buttons

• [ ] pybricks: LEGO hub with Bluetooth Gamepad demo app demo (with the gamepad tester to test the hw first)

  • [ ] Pybricks MicroPython LEGO Bluetooth BLE demo build with https://trevorsandy.github.io/lpub3d/ ?

[nkarstens]

@laurensvalk @dlech Do you have any ideas or requirements for USB support in case we wanted to try to implement and submit a pull request? I'm running an FLL team and would like to avoid Bluetooth at competitions.

[lawrencet149]

@laurensvalk @dlech Do you have any ideas or requirements for USB support in case we wanted to try to implement and submit a pull request? I'm running an FLL team and would like to avoid Bluetooth at competitions.

@nkarstens I am willing to help with this effort if you need the help.

[dlech]

Ideas:

• It should work on Windows with no special drivers. See this page for what Windows expects for this.
• It should use a vendor-specific device class (0xFF).
• It should wrap the Pybricks Profile that we use for BLE. This includes emulating the GATT protocol to some extent.

[westurner]

FWIW also there are pylgbst and ipylgbst.

pylgbst wraps various Python Bluetooth backends for various OS; https://github.com/undera/pylgbst#bluetooth-backend-prerequisites :

Windows users may first turn to the Bleak backend, which should support any internal or external Bluetooth adapter recognized by the OS. The Windows version of pygatt backend will only work with a Bluegiga BLED112 Bluetooth Smart Dongle.

• https://github.com/undera/pylgbst/blob/master/pylgbst/comms/cbleak.py https://github.com/undera/pylgbst/blob/master/tests/test_cbleak.py
• https://github.com/undera/pylgbst/blob/master/pylgbst/comms/cbluepy.py
• https://github.com/undera/pylgbst/blob/master/pylgbst/comms/cgatt.py
• https://github.com/undera/pylgbst/blob/master/pylgbst/comms/cgattlib.py
• https://github.com/undera/pylgbst/blob/master/pylgbst/comms/cpygatt.py

ipylgbst wraps WebBluetooth for use as a JupyterLab extension https://github.com/jupyter-robotics/ipylgbst:

• https://github.com/jupyter-robotics/ipylgbst/blob/main/examples/introduction.ipynb
• https://github.com/undera/pylgbst/blob/master/examples/vernie/android_remote.py
  • https://play.google.com/store/apps/details?id=com.mscino.sensornode

[nkarstens]

All right I've looked into this a little bit and I'm hoping I'm not too far out of my depth here...

@dlech It looks like there was support for a USB serial connection at one point (usb_stm32_serial.c), but that was removed. What can you tell me about the history there? It looks like you suggested WinUSB; would a serial protocol be a better option for wrapping the GATT profile? Do you have any documentation on options for debugging firmware (such as JTAG or a printf)?

@lawrencet149 Thanks for the offer, do you know much about firmware development?

[dlech]

Since BLE is message-based instead of stream-based, I don't think serial is a good fit. Also we don't want the official LEGO apps to "see" devices that are running Pybricks firmware.

The initial USB serial implementation was just to debug before we got Bluetooth working.

Debugging is a bit tricky since there is no JTAG (lock bit was set at the factory, so it doesn't work) and printf has to be used with much caution. If we really need to, we can hijack one of the I/O ports for printing via UART. This is discussed in a bit more detail at https://github.com/pybricks/pybricks-micropython/pull/80#issuecomment-1008333124.

[nkarstens]

@dlech I got a NUCLEO-F413ZH board to help with prototyping the initial USB work and will then plan to migrate to pybricks-micropython from there.

It looks like we will need a USB vendor ID and product ID. pid.codes does subassignments of their vendor ID, 1209, for open source projects that meet their criteria. Can you please review the prerequisites listed here and let me know if this sounds good to you? If so, I will take care of making a pull request and send to you for approval, or you can do that yourself and let me know when it has been merged in.

[dlech]

We should use the LEGO USB VID/PIDs that are already assigned to these hubs. https://github.com/pybricks/technical-info/blob/master/assigned-numbers.md#usb-device-ids

[nkarstens]

I was under the impression that VID and PID help determine which driver is loaded, which may cause conflict with the factory firmware (see https://unix.stackexchange.com/questions/391919/do-vendor-id-and-product-id-alone-determine-the-driver-used-for-a-usb-device for one discussion about this). In that case, using a unique set of IDs would be preferable. What do you think?

Nate

On Thu, Sep 28, 2023, 08:05 David Lechner @.***> wrote:

We should use the LEGO USB VID/PIDs that are already assigned to these hubs. https://github.com/pybricks/technical-info/blob/master/assigned-numbers.md#usb-device-ids

— Reply to this email directly, view it on GitHub https://github.com/pybricks/support/issues/262#issuecomment-1739126458, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGKEWXQU4RH65EZKYIH22Q3X4VY3LANCNFSM4YIEEWRQ . You are receiving this because you were mentioned.Message ID: @.***>

[dlech]

They can be, but for the SPIKE and MINDSTORMS hubs, there is no special driver. It is using the class/subclass/protocol to match to a driver instead. The official LEGO firmware is using the standard serial port (CDC ACM) class/subclass/protocol. If we are using a vendor-specific class/subclass/protocol, then there shouldn't be any conflict.

[nkarstens]

@dlech I got it to a point where Windows will load a WinUSB driver for it. Please see the draft patch at pybricks/pybricks-micropython#208 and let me know if you have any suggestions. Thanks!

[dlech]

Nice! Looking forward to see what you came up with.

[nkarstens]

Just FYI for anyone monitoring this issue. The following pull requests have enough USB support implemented to push a program:

• pybricks/pybricks-micropython#208
• pybricks/pybricksdev#69

@dlech has incorporated these into a dlech-usb branch in each project and is making additional improvements.