search

adafruit / Adafruit_CircuitPython_BLE

Bluetooth Low Energy (BLE) library for CircuitPython
MIT License
127 stars 58 forks source link

VoiceOver Modifier keys not recognised by iOS device over BLE HID #55

Closed jscholes closed 4 years ago

jscholes commented 4 years ago

When sending keystrokes to an iOS device using the new BLE HID support, any modifiers seem to be discarded. For example, using the following code after a connection has been established (where kbd is an instance of adafruit_hid.keyboard.Keyboard):

kbd.send(Keycode.CONTROL, Keycode.OPTION, Keycode.H)

With iOS VoiceOver turned on, Ctrl+Option+H should activate the Home button, showing the home screen. But VoiceOver is only reporting that the H key has been received.

On Android, I turned on the TalkBack screen reader, and then did:

kbd.send(Keycode.CONTROL, Keycode.ALT, Keycode.H)

This is likewise intended as a keystroke to activate the Home button, and on Android it worked as expected. Does iOS require keyboard reports to be constructed in a nonstandard format?

CC @dhalbert and @sinabahram

dhalbert commented 4 years ago

Keycode.SHIFT is working (I'm getting capital letters), so I don't think it's modifier keys in general.

I did a websearch for [ iOS go to home screen keyboard shortcut ], and what I saw says to use Command+H, i.e. kbd.send(Keycode.COMMAND, Keycode.H) E.g. https://www.howtogeek.com/398356/20-keyboard-shortcuts-every-ipad-owner-should-know/

Could you try that?

jscholes commented 4 years ago

@dhalbert sure, and thanks for the response. I'll try this in a short while here.

I should note that Ctrl+Option+H should absolutely work, as it does on a physical keyboard. But only with VoiceOver turned on. There are a bunch of other commands provided by VoiceOver, which won't have system-level equivalents.

jscholes commented 4 years ago

Cmd+H does work.

dhalbert commented 4 years ago

I see https://support.apple.com/guide/iphone/control-voiceover-with-magic-keyboard-iph6c494dc6/ios, and that "VO" is either Ctrl+Option or Shift-Lock. I don't know what to say, but the code for BLE HID is very similar at the lower level. The BLE part of it doesn't even know about modifier keys specially: it is just passing on the modifier bits generated by adafruit_hid. Perhaps the low-level keycodes are actually different. Are you using a Magic keyboard? What an Apple BLE keyboard sends is actually not well documented, if I recall correctly. I don't have an Apple BLE keyboard at hand at the moment to spy on.

jscholes commented 4 years ago

I have used an Apple keyboard, but these commands have also worked with Bluetooth keyboards from other manufacturers. We can reach out to a contact at Apple, and try to spy on what an Apple keyboard is doing. But like you, I'm stumped at this point.

dhalbert commented 4 years ago

@jscholes We're reviewing all our open issues for action. Have you reached any further understanding on this? Thanks.

jscholes commented 4 years ago

@dhalbert Sorry for not updating sooner. We got this working by sending each modifier one by one in a particular order, and then releasing one-by-one with that order reversed. Option then Control, if it helps anybody coming to this thread later.