search

unfoldedcircle / feature-and-bug-tracker

Feature and bug tracker repository for Unfolded Circle products
25 stars 0 forks source link

Pair as Bluetooth keyboard to devices #65

Open martonborzak opened 1 year ago

martonborzak commented 1 year ago

Initial Bluetooth feature set:

Further features might be introduced later, for example a virtual gamepad or mouse. Supporting a native manufacturers' Bluetooth remote is mostly not possible since they are almost always proprietary without public information. If there are manufacturers with open specifications, we will investigate it, but now the focus is providing a HID keyboard.

Update 2024-07-10: Beta firmware v1.8.0 is out

This includes the first BLE keyboard & mouse peripheral support 🎉 Documentation is in the core-api repository: https://github.com/unfoldedcircle/core-api/tree/main/doc/bt

Please provide feedback in the Discord software/beta-feedback channel, or in this GitHub issue.

Freshhat commented 1 year ago

Hi Marton, i'm just asking for a ETA for this topic.

I see that you have added this feature to the Roadmap Q4 2023 milestone. For me this is the last topic, which is blocking me to use the remote as a daily device. In the initial Roadmap planning it was scheduled for October 2023:

image

zehnm commented 7 months ago

It is time for an update since this feature has been moved back in our roadmap multiple times. Unfortunately the Bluetooth functionality won't be ready in Q1.

The reasons are manyfold. It’s not just a complex technology, but also involves dealing with proprietary chipset firmware and vendor dependent support and bug fixes, which has to play together with a software Bluetooth stack. Even if that all works together, it can still fail with vendor-interoperability issues or bad user experience. Ideally, a keyboard peripheral must work with every device supporting a Bluetooth keyboard. Discoverability and reconnection must be fast and automatic. This has taken way more effort than initially planned, especially the Linux Bluetooth stack has given us the blues.

After many failed attempts, we came to the realization that we were on the wrong path and that a different Bluetooth stack would solve many problems. We’ve been fighting with getting virtual peripheral devices working. That is: simulating a keyboard, mouse, or gamepad with the Remote Two.

Now the good news: we are in the final stage of evaluating alternative Bluetooth stacks and have a working prototype of a BLE keyboard, which works with Apple and Android devices. We still have to test a few things, software wise and also in terms of interoperability, but we are confident that we can soon commit on a new Bluetooth stack and move forward with the Bluetooth functionality.

Freshhat commented 7 months ago

Thank you very much for the update Markus. For the moment the missing Bluetooth integration is still the blocker to really use the remote in a daily use. In my setup i have a few fire tv sticks, which for sure can be also controlled via Android Debug Bridge inside Home Assistant, but there the performance is very weak.

So if we from community side can support in any way, please just give us a hands up, would also have multiple OS types in my Home Lab where i could test against.

tiki-ftl commented 7 months ago

Thank you for the update! Since the BT functionality still seems the only way to control a PC without extra hardware, I very much welcome this, indeed, I am "dependent" on it. Is it possible, to name the remote or it`s BT-device, respectively, with an unique and clear name? Please do not use "HID-device" like the Harmony did. Thank you!

Freshhat commented 5 months ago

Is there any update on this functionality? The last update from UC Team is over 1 month ago

zehnm commented 5 months ago

Is it possible, to name the remote or it`s BT-device, respectively, with an unique and clear name? Please do not use "HID-device" like the Harmony did.

Currently it's named RemoteTwo xxxxxx where x = last 6 digits of the MAC address. Maybe we have to shorten it further, if we have to send more advertising data. We are not sure yet if we make it user-editable. Bluetooth device names are quite tricky, especially if they can be changed. Most devices cache the initially discovered name associated with the BT address. This can get confusing when the new name doesn't show up on the device which is trying to pair with the remote. Most of the time, the correct name is only shown after pairing, toggling bluetooth, or a reboot.

zehnm commented 5 months ago

Time for an update! This endeavour has become a deep dive into Bluetooth LE, multi-device connectivity and USB HID report descriptor / HID usage table "madness". Just the HID usage tables alone was a whole new project and I'd have enough material for multiple blog posts :-) I've definitely grown more grey hairs during the last two months getting it to work! Now we have to test the multiple hundred key-codes, and figure out which ones work with which device. A Bluetooth keyboard is not as simple as it seems first (besides simple keys A-Z, 0-1, etc). Even though the standard key codes already define multiple media control commands, but they rarely work with a device (Android tablets being the exception). One has to resort to the "generic consumer control device", which is another report descriptor the keyboard has to send to the host. This won't hold up the release, we'll start with the most common key codes, similar to a physical BT keyboard, and keep on testing and adding commands and pre-configured "device profiles" (similar to what we've started in the Android TV integration).

Current state: we have a working end-to-end proof of concept 🥳

Timeline:

zehnm commented 4 months ago

Progress update:

Testing various devices and figuring out working keys is quite a time intensive task. It also revealed that we have to add more features after the initial release. An "IR-sidekick" would be handy for devices which can only be powered on by infrared (for example LG TVs). This could be achieved manually with an activity, but including it directly in the device specific entity would be more convenient. We’ll think about it, first things first. The tested devices and derived device-profiles are also published in the above branch (under doc/bt). This should be quite helpful as a starting point for custom devices or to customise an official profile. How, you may ask? Device profiles can be uploaded as a resource with the REST Core-API! Support in the web-configurator might not yet be included in the initial release, but is in the backlog.

JackJPowell commented 4 months ago

This sounds like a ton of work @zehnm! And really exciting that it’s almost here. Is there a quick guide some of us could follow to begin testing other devices and building their device profiles? I know a lot of people are awaiting PlayStation support and I wouldn’t mind helping if I knew how to start. No matter what, thanks for all your hard work!

zehnm commented 4 months ago

Is there a quick guide some of us could follow to begin testing other devices and building their device profiles?

Easiest way to start finding the basic commands is to use a Bluetooth LE keyboard, pair it with your target device and start pressing keys. Every manufacturer does it a bit differently, some use the F1..10 keys for media commands, others use the defined HID consumer codes (most BT keyboards support a few media playback buttons), or abuse arbitrary usage codes. Testing with a physical keyboard should give the main navigation functionality and hopefully volume control and some additional shortcuts like opening menus etc.. The rest needs to be tested with the REST-API on the Remote. Some technical information is already in the BT branch. But this requires USB HID usage table knowhow.

Temporary links until the branch is merged (around next week):

PlayStation support

Still working on it. All tested devices worked so far, except PS5. Since I don't own one, remote testing takes a bit longer.

kennymc-c commented 4 months ago

How long does it take for remote the send a command to a Bluetooth device when the remote wakes up from deep sleep? For ip integrations this is quite long as the remote has to reconnect to the wifi network. Will this be faster with Bluetooth?

WillTriumph1 commented 4 months ago

Amy idea is newer Samsung TV remotes or older generation than 4 Gen AppleTV Bluetooth will work.

Pebowski commented 4 months ago

Will it also work with Amazon Fire TV stick?

zehnm commented 4 months ago

How long does it take for remote the send a command to a Bluetooth device when the remote wakes up from deep sleep?

At the moment the BT controller is fully powered off in standby and re-initialised when waking up, to be sure everything is working correctly. Low-power BT controller modes are vendor specific and require a bit of work. To not loose more time releasing the BT functionality we decided to go the safe route in the beginning. We will definitely look further into low-power modes when the main BT functionality is released and working.

With Bluetooth LE, the connection is initiated from the central, not the peripheral (BT terms: peripheral = usually a low power device offering & advertising a service (e.g. a keyboard or mouse), central = device connecting to a peripheral). This means it also depends on how fast the other devices reconnect once they see the Remote advertising again.

A quick preliminary test with Apple TV and Samsung M7 showed that it takes close to 2 seconds from wakeup to ready (connection established & subscribed to GATT service updates). Breakdown:

Shaving off one second seems realistic if we get low-power mode working and avoid a cold start with loading the firmware. Again, this was just a quick test and numbers vary depending on system load and number of bluetooth devices.

Amy idea is newer Samsung TV remotes or older generation than 4 Gen AppleTV Bluetooth will work.

If it supports pairing with a Bluetooth LE keyboard, it should work. Devices released after 2015 have a good chance of supporting BLE (Bluetooth 4 and newer). According to https://en.wikipedia.org/wiki/Apple_TV#Technical_specifications the 3rd gen Apple TV has Bluetooth 4, but I'm not sure it also supports BLE keyboards. I might have a 2nd or 3rd gen ATV in the cellar, I’ll check…

Will it also work with Amazon Fire TV stick?

If it supports Bluetooth LE keyboard pairing, then it will most likely work. It also depends if it can be controlled by a keyboard and if all important functions are supported. We have ordered a Fire TV stick and should arrive soon to start testing.

zehnm commented 4 months ago

Amy idea is newer Samsung TV remotes or older generation than 4 Gen AppleTV Bluetooth will work.

Found a 3rd gen Apple TV model from 2012: sorry, doesn't work. (It also doesn't find a Logitech Craft BLE keyboard).

Pebowski commented 4 months ago

CleanShot 2024-06-18 at 00 09 30@2x I could pair a Logitech K360 BT keyboard to the Fire TV stick and the keyboard works. Only für those keys with the red circle around I found no equivalent on the keyboard.

zehnm commented 4 months ago

Only für those keys with the red circle around I found no equivalent on the keyboard.

Good to hear that the keyboard works. The missing keys should be easy to find when have the device in our hands. Volume & mute for example has 3 different usage codes, one of them will certainly work.

mchangsp commented 4 months ago

At the moment the BT controller is fully powered off in standby and re-initialised when waking up, to be sure everything is working correctly. Low-power BT controller modes are vendor specific and require a bit of work. To not loose more time releasing the BT functionality we decided to go the safe route in the beginning. We will definitely look further into low-power modes when the main BT functionality is released and working.

What if you use a ESPHome device to maintain the Bluetooth connection? When the Remote Two wakes up it sends commands to the ESPHome device over TCP/IP. The ESPHome device is powered up all the time so it can forward the command immediately because it maintains the Bluetooth connection. The ESPHome device can be put in a place near the device intended to be contolled over Bluetooth.

https://esphome.io/components/ble_client.html#

martonborzak commented 4 months ago

We have been testing the Bluetooth functionality and were about to release the public beta, when we have discovered a weird behaviour with deep sleep. We're working hard to track this down, but it's not a straight forward one as it does not show up on all the remotes we have tested. This will delay the public release a bit as we don't want to push out something that we know breaking a working functionality.

zehnm commented 4 months ago

Short update: after over a week of digging we've found the issue with deep sleep! The fix will take another day or two, then we can finally release it as beta. ETA: next week.