Add some BLE/Arduino support to Creator/Paper Playground

[jessegreenberg]

Notes from a first connection test with a BLE enabled arduino. I am using an Uno R4 with wifi and BLE support.

1) Using this .ino file, which enables BLE support on the device. It draws a heart on the on-device LEDs when the characteristic value is non-zero, and turns off the LEDs when the value is zero.

```ino /* LED This example creates a Bluetooth® Low Energy peripheral with service that contains a characteristic to control an LED. The circuit: - Arduino MKR WiFi 1010, Arduino Uno WiFi Rev2 board, Arduino Nano 33 IoT, Arduino Nano 33 BLE, or Arduino Nano 33 BLE Sense board. You can use a generic Bluetooth® Low Energy central app, like LightBlue (iOS and Android) or nRF Connect (Android), to interact with the services and characteristics created in this sketch. This example code is in the public domain. */ #include #include "Arduino_LED_Matrix.h" // Include the LED_Matrix library BLEService ledService("19B10000-E8F2-537E-4F6C-D104768A1214"); // Bluetooth® Low Energy LED Service // Bluetooth® Low Energy LED Switch Characteristic - custom 128-bit UUID, read and writable by central BLEByteCharacteristic switchCharacteristic("19B10001-E8F2-537E-4F6C-D104768A1214", BLERead | BLEWrite); const int ledPin = LED_BUILTIN; // pin to use for the LED // Global declaration of the 'heart' array const uint32_t heart[] = { 0x3184a444, 0x44042081, 0x100a0040 }; const uint32_t off[] = { 0x00000000, 0x00000000, 0x00000000 }; ArduinoLEDMatrix matrix; // Create an instance of the ArduinoLEDMatrix class void setup() { Serial.begin(9600); matrix.begin(); // Initialize the LED matrix while (!Serial); // set LED pin to output mode pinMode(ledPin, OUTPUT); // begin initialization if (!BLE.begin()) { Serial.println("starting Bluetooth® Low Energy module failed!"); while (1); } // set advertised local name and service UUID: BLE.setLocalName("HEART"); BLE.setAdvertisedService(ledService); // add the characteristic to the service ledService.addCharacteristic(switchCharacteristic); // add service BLE.addService(ledService); // set the initial value for the characeristic: switchCharacteristic.writeValue(0); // start advertising BLE.advertise(); Serial.println("BLE LED Peripheral"); } void loop() { // listen for Bluetooth® Low Energy peripherals to connect: BLEDevice central = BLE.central(); // if a central is connected to peripheral: if (central) { Serial.print("Connected to central: "); // print the central's MAC address: Serial.println(central.address()); // while the central is still connected to peripheral: while (central.connected()) { // if the remote device wrote to the characteristic, // use the value to control the LED: if (switchCharacteristic.written()) { if (switchCharacteristic.value()) { // any value other than 0 // Serial.println("LED on"); // digitalWrite(ledPin, HIGH); // will turn the LED on matrix.loadFrame(heart); } else { // a 0 value // Serial.println(F("LED off")); matrix.loadFrame(off); // digitalWrite(ledPin, LOW); // will turn the LED off } } } // when the central disconnects, print it out: Serial.print(F("Disconnected from central: ")); Serial.println(central.address()); } } ```

2) I added a button to the Interactive Display page to establish a BLE connection.


And here is the connection button in the Interactive Board page:

```jsx { // Request the device for a Bluetooth connection navigator.bluetooth.requestDevice( { acceptAllDevices: true } ) .then( device => { console.log( 'Connecting to device...', device ); return device.gatt.connect(); // Connect to GATT Server } ) .then( server => { console.log( 'Getting LED Service...' ); return server.getPrimaryService( '19b10000-e8f2-537e-4f6c-d104768a1214' ); // Get LED Service } ) .then( service => { console.log( 'Getting LED Characteristic...' ); return service.getCharacteristic( '19b10001-e8f2-537e-4f6c-d104768a1214' ); // LED Characteristic UUID } ) .then( characteristic => { console.log( 'LED Characteristic is ready!' ); window.ledCharacteristic = characteristic; // Make characteristic globally available // document.querySelector( '#toggleLed' ).disabled = false; } ) .catch( error => { console.error( 'Connection failed', error ); } ); }} >Connect to BLE ```

3) This Custom Code was added to a program in Creator to control the BLE characteristic value from the paper:

In `onProgramAdded`: ```js if ( window.ledCharacteristic ) { const data = new Uint8Array([1]); // Data to turn the LED on, use 0 to turn off window.ledCharacteristic.writeValue(data) .then(() => { phet.paperLand.console.log( 'Toggled LED!' ); }) .catch(error => { phet.paperLand.console.error('Toggle failed', error); }); } ``` In `onProgramRemoved` ```js if ( window.ledCharacteristic ) { const data = new Uint8Array([0]); // Data to turn the LED on, use 0 to turn off window.ledCharacteristic.writeValue(data) .then(() => { phet.paperLand.console.log( 'Toggled LED!' ); }) .catch(error => { phet.paperLand.console.error('Toggle failed', error); }); } ```

In order for the browser to connect, I had to enable bluetooth devices in Windows settings. Otherwise, this was pretty much plug and play.

[jessegreenberg]

Also trying out the micro:bit. It took a few more steps to set up (just to connect on Windows, not creator/paper-playground).

1) I needed a specific usb-micro coord for windows to connect to the device. All powered the device, but only some allowed connection to upload code. 2) Go through these steps to pair the device to system BLE: https://makecode.microbit.org/v0/reference/bluetooth/bluetooth-pairing. (Press A and B buttons at the same time, then press "Reset" button. Then find device in system settings). 3) Create a new project in Makecode. Go through settings to enable BLE extension (requires you to disable radio extension) 4) In MakeCode (https://makecode.microbit.org/#editor), this sets up the BLE service and then draws a heart on the LEDs when there is a successful connection.

bluetooth.startLEDService()

basic.showString("Hello!")
basic.pause(2000)
basic.clearScreen()

bluetooth.onBluetoothConnected(function() {
    basic.showIcon(IconNames.Heart)
})

I am having a tough time getting the actual characteristic values to read/write from the abstracted makecode functions. Its possible examples here may have enough info to find what we need: https://makecode.microbit.org/reference/bluetooth. But I haven't been able to figure it out yet.

I tried these characteristic values from the source code just in case, but they didn't work. https://github.com/thegecko/microbit-web-bluetooth/blob/master/src/services/led.ts#L30C1-L56C65

And also these: https://github.com/thegecko/microbit-web-bluetooth/blob/master/src/services/uart.ts#L32C1-L36C1

I got an error: "PaperLandControls.js:190 Connection failed DOMException: No Services matching UUID 6e400002-b5a3-f393-e0a9-e50e24dcca9e found in Device."

I also found this documentation about the BLE characteristics: https://lancaster-university.github.io/microbit-docs/resources/bluetooth/bluetooth_profile.html

through this StackOverflow thread: https://stackoverflow.com/questions/63367507/how-can-i-access-the-light-sensor-value-when-using-bbcmicrobit-over-bluetooth

And got the same errors.

However, switching to this code gets it working. In this example, adding a paper program turns on or off the top right LED in the matrix on the front of the chip.

Here is the MakeCode JavaScript:

```js bluetooth.startUartService() bluetooth.startLEDService(); basic.showString("Hello!") basic.clearScreen() bluetooth.onBluetoothConnected(function () { basic.showIcon(IconNames.Heart) basic.pause(2000) basic.clearScreen() }) bluetooth.onUartDataReceived(serial.delimiters(Delimiters.NewLine), function () { basic.showString(bluetooth.uartReadUntil(serial.delimiters(Delimiters.Space))) }) basic.forever(function() { bluetooth.uartWriteNumber(input.lightLevel()); basic.pause( 500 ); }) ```

With these characteristic IDs:

```jsx { // Request the device for a Bluetooth connection navigator.bluetooth.requestDevice( { acceptAllDevices: true, // for the arduino // optionalServices: [ '19b10000-e8f2-537e-4f6c-d104768a1214' ] // for the microbit optionalServices: [ 'e95dd91d-251d-470a-a062-fa1922dfa9a8' ], name: 'Micro:bit' } ) .then( device => { console.log( 'Connecting to device...', device ); return device.gatt.connect(); // Connect to GATT Server } ) .then( server => { console.log( 'Getting LED Service...' ); // for ardunio // return server.getPrimaryService( '19b10000-e8f2-537e-4f6c-d104768a1214' ); // Get LED Service // for microbit return server.getPrimaryService( 'e95dd91d-251d-470a-a062-fa1922dfa9a8' ); // Get LED Service } ) .then( service => { console.log( 'Getting LED Characteristic...' ); // for arduino // return service.getCharacteristic( '19b10001-e8f2-537e-4f6c-d104768a1214' ); // LED Characteristic UUID // for microbit return service.getCharacteristic( 'e95d7b77-251d-470a-a062-fa1922dfa9a8' ); // LED Characteristic UUID } ) .then( characteristic => { console.log( 'LED Characteristic is ready!' ); window.ledCharacteristic = characteristic; // Make characteristic globally available // document.querySelector( '#toggleLed' ).disabled = false; } ) .catch( error => { console.error( 'Connection failed', error ); } ); }} >Connect to BLE ```

And this code in the onProgramAdded

```js if (window.ledCharacteristic) { window.ledCharacteristic.readValue() .then(value => { // value is a DataView const currentValue = value.getUint8(0); // Assuming the value is a single byte phet.paperLand.console.log('Current LED Value:', currentValue); // Optionally, you can toggle the LED based on the currentValue // This part assumes you want to toggle the LED's state; remove it if not needed const dataToWrite = new Uint8Array([currentValue === 0 ? 1 : 0]); return window.ledCharacteristic.writeValue(dataToWrite); }) .then(() => { phet.paperLand.console.log('Toggled LED based on current value!'); }) .catch(error => { phet.paperLand.console.error('Reading or toggling failed', error); }); } phet.paperLand.console.log('ADDING'); ```

While testing, it failed a couple of times for unknown reasons, but often worked. It is important to refresh the board page whenever you make a change in creator. Otherwise the GATT request fails. I don't know why.

I also don't know why this only controls the top right LED, or how you would control others.

Overall, it was a lot harder to find micro:bit documentation for anything outside of basic MakeCode examples.

[jessegreenberg]

THe BLE button is in branch ble-test.

[brettfiedler]

Listing some resources I found:

https://makecode.microbit.org/reference/bluetooth

https://lancaster-university.github.io/microbit-docs/ble/uart-service/

https://github.com/antefact/microBit.js/blob/master/src/microBit.js

(serial) https://makecode.microbit.org/device/serial

Noting that "Radio" appears to be specifically for multiple microbits communicating to each other. All the documentation there is not too helpful.

[brettfiedler]

attempt to add temperature service

```diff diff --git a/client/board/PaperLandControls.js b/client/board/PaperLandControls.js index 1f4d01f..095e611 100644 --- a/client/board/PaperLandControls.js +++ b/client/board/PaperLandControls.js @@ -19,47 +19,47 @@ const MIN_REMOVAL_DELAY = 0; const MAX_REMOVAL_DELAY = 5; const REMOVAL_INTERVAL_STEP = 0.5; -export default function PaperLandControls( props ) { - const [ positionInterval, setPositionInterval ] = useState( props.initialPositionInterval ); - const [ removalDelay, setRemovalDelay ] = useState( props.initialRemovalDelay ); - const [ consoleVisible, setConsoleVisible ] = useState( true ); - const [ printSpeechSynthesis, setPrintSpeechSynthesis ] = useState( false ); +export default function PaperLandControls(props) { + const [positionInterval, setPositionInterval] = useState(props.initialPositionInterval); + const [removalDelay, setRemovalDelay] = useState(props.initialRemovalDelay); + const [consoleVisible, setConsoleVisible] = useState(true); + const [printSpeechSynthesis, setPrintSpeechSynthesis] = useState(false); // A reference to the resize observer which will be created when we enter full-screen, and removed when we exit - const resizeObserverRef = useRef( null ); + const resizeObserverRef = useRef(null); - useEffect( () => { + useEffect(() => { const fullScreenListener = fullScreen => { - if ( props.sceneryDisplay ) { - if ( fullScreen ) { + if (props.sceneryDisplay) { + if (fullScreen) { // remove the styling that positions the board for development - props.sceneryDisplay.domElement.classList.remove( styles.simDisplayPanel ); - props.sceneryDisplay.domElement.classList.remove( styles.boardPanel ); + props.sceneryDisplay.domElement.classList.remove(styles.simDisplayPanel); + props.sceneryDisplay.domElement.classList.remove(styles.boardPanel); // If we still have a resize observer, make sure it is disconnected - if ( resizeObserverRef.current ) { + if (resizeObserverRef.current) { resizeObserverRef.current.disconnect(); } // There is a delay between when the resize happens and when the values for window dimensions are updated. // Waiting until the resize event confirms that we will set the scenery display and paper-land display // size to accurate values. - resizeObserverRef.current = new ResizeObserver( entries => { - for ( const entry of entries ) { + resizeObserverRef.current = new ResizeObserver(entries => { + for (const entry of entries) { // Assumes that we are observing the full-screen element (the Display domElement) const { width, height } = entry.contentRect; - props.sceneryDisplay.setWidthHeight( width, height ); - phet.paperLand.displaySizeProperty.value = new phet.dot.Dimension2( width, height ); + props.sceneryDisplay.setWidthHeight(width, height); + phet.paperLand.displaySizeProperty.value = new phet.dot.Dimension2(width, height); } - } ); - resizeObserverRef.current.observe( props.sceneryDisplay.domElement ); + }); + resizeObserverRef.current.observe(props.sceneryDisplay.domElement); } else { // We are done with the resize observer, disconnect it - if ( resizeObserverRef.current ) { + if (resizeObserverRef.current) { resizeObserverRef.current.disconnect(); resizeObserverRef.current = null; } @@ -68,35 +68,35 @@ export default function PaperLandControls( props ) { const smallHeight = 480; // re-apply styling for development - props.sceneryDisplay.domElement.classList.add( styles.simDisplayPanel ); - props.sceneryDisplay.domElement.classList.add( styles.boardPanel ); - props.sceneryDisplay.setWidthHeight( smallWidth, smallHeight ); - phet.paperLand.displaySizeProperty.value = new phet.dot.Dimension2( smallWidth, smallHeight ); + props.sceneryDisplay.domElement.classList.add(styles.simDisplayPanel); + props.sceneryDisplay.domElement.classList.add(styles.boardPanel); + props.sceneryDisplay.setWidthHeight(smallWidth, smallHeight); + phet.paperLand.displaySizeProperty.value = new phet.dot.Dimension2(smallWidth, smallHeight); } } }; - phet.scenery.FullScreen.isFullScreenProperty.link( fullScreenListener ); + phet.scenery.FullScreen.isFullScreenProperty.link(fullScreenListener); // cleanup, removing the listener before re-render return () => { - phet.scenery.FullScreen.isFullScreenProperty.unlink( fullScreenListener ); + phet.scenery.FullScreen.isFullScreenProperty.unlink(fullScreenListener); }; - }, [ props.sceneryDisplay ] ); + }, [props.sceneryDisplay]); // Print speech synthesis to the console - useEffect( () => { + useEffect(() => { const printListener = response => { - if ( printSpeechSynthesis ) { - phet.paperLand.console.log( 'Speech', `"${response}"` ); + if (printSpeechSynthesis) { + phet.paperLand.console.log('Speech', `"${response}"`); } }; - phet.scenery.voicingManager.startSpeakingEmitter.addListener( printListener ); + phet.scenery.voicingManager.startSpeakingEmitter.addListener(printListener); return () => { - phet.scenery.voicingManager.startSpeakingEmitter.removeListener( printListener ); + phet.scenery.voicingManager.startSpeakingEmitter.removeListener(printListener); }; - }, [ printSpeechSynthesis ] ); + }, [printSpeechSynthesis]); return (

@@ -112,8 +112,8 @@ export default function PaperLandControls( props ) { value={positionInterval} onChange={event => { const newValue = event.target.value; - setPositionInterval( newValue ); - props.updatePositionInterval( newValue ); + setPositionInterval(newValue); + props.updatePositionInterval(newValue); }} />

@@ -130,8 +130,8 @@ export default function PaperLandControls( props ) { value={removalDelay} onChange={event => { const newValue = event.target.value; - setRemovalDelay( newValue ); - props.updateRemovalDelay( newValue ); + setRemovalDelay(newValue); + props.updateRemovalDelay(newValue); }} />

[jessegreenberg]

One thing I found today is that on an insecure origin (like localhost), you must declare all services you want to connect to in optionalServies:

           // Request the device for a Bluetooth connection
            navigator.bluetooth.requestDevice( {
              acceptAllDevices: true,

              // Putting the services here lets us connect even on an insecure origin
              optionalServices: [ 'e95d6100-251d-470a-a062-fa1922dfa9a8' ]
            } )

For micro: bit we can declare everything. But for Arduino and others, we may need to have the user enter serice IDs twice - once in Creator, and once in Interactive Display, which is a bummer.

[jessegreenberg]

Pretty good progress on this today in the dev branch. We can observe a characteristic change from generated code, after the user has provided the characteristic and service IDs they are interested in.

For next time:

Actually connect to Creator components by continuinig to fill in this part of the code template: ```js BluetoothListenerComponent: { onProgramAdded: ` phet.paperLand.boardBluetoothServers.addCharacteristicListener( '{{SERVICE_ID}}', '{{CHARACTERISTIC_ID}}', value => { // TODO: This is where we would bring in component references, control functions, // and finally call the user's control function. phet.paperLand.console.log( 'Value:', value.getUint8( 0 ) ); } ) .then( addedListener => { scratchpad.characteristicListener = addedListener; } ) .catch( error => { phet.paperLand.console.error( error ); } ); `, onProgramRemoved: ` phet.paperLand.console.log( 'Removing a BLE component' ); phet.paperLand.boardBluetoothServers.removeCharacteristicListener( '{{SERVICE_ID}}', '{{CHARACTERISTIC_ID}}', scratchpad.characteristicListener ).catch( error => { phet.paperLand.console.error( error ); } ); ` } ```

[jessegreenberg]

Some documentation and examples. Also information about memory pitfalls. https://lancaster-university.github.io/microbit-docs/ble/uart-service/#example-microbit-application-animal-vegetable-mineral-game

[jessegreenberg]

We want to try UART messages. I added that service/characteristics for it, but we didn't verify that it works.