Keyboard passthrough that works with TalkBack

[jha11y]

• [ X] I have checked that a similar feature request does not already exist.

Is your feature request related to a problem? Please describe. I am testing built in Screen reader on the device (TalkBack) which has specific keyboard shortcuts (e.g., Alt+ arrow right for next element). Pressing this combination does not work for Native apps and in Chrome.

Describe the solution you'd like Allow the keyboard keypresses [Alt, Alt+letter, etc.] (I'm running windows 10 and usb keyboard) to be appropriately sent to the Android device so the work with TalkBack

Describe alternatives you've considered I can use a Bluetooth keyboard for testing as these commands work with Bluetooth keyboards, but this would be helpful for instructing other who may not have a Bluetooth keyboard

[rom1v]

With #1598, you could forward Ctrl and Shift. Alt is used (by default) for scrcpy shortcuts.

Forwarding Alt would cause unexpected behaviors anyway: https://github.com/Genymobile/scrcpy/pull/1465#issuecomment-638106393

[jha11y]

Options for TalkBack keyboard shortcut modifier key is ALT, Search (CMD/Windows Key), or ALT+ SHIFT. So options are limited. Looks like this may not be an option. I'll try the version you mentioned in the comment, but I thought I had tried that one already.

[pbsinnett]

Hello,

I am a blind Android user and would also really love to get TalkBack controls working. Even when I switch TalkBack's key map to classic and reassign keys to something like control plus something, it doesn't work. A requirement for the TalkBack shortcuts is that it has to contain a control or alt key if using the classic key map, or to use control or the search key (not sure what that would be on Windows) for the standard key map.

I'm not sure what a solution might be, but this is currently the only thing that keeps me from using this great program!

[rom1v]

I didn't know TalkBack, so I took a look. Indeed, it seems to require the Alt key.

So as a quick test, I forwarded the Alt key.

In "classic key map", I can edit the shortcut, it correctly recognizes the Alt key to customize the shortcut.

However, pressing the shortcut does not work (it is not interpreted by TalkBack, it does not execute the action).

Even when I switch TalkBack's key map to classic and reassign keys to something like control plus something, it doesn't work.

So it seems that the Alt key is not the only issue. I guess it works at a lower level, so injecting keys via the InputManager does not trigger TalkBack shortcuts.

source code diff

```diff diff --git a/app/src/event_converter.c b/app/src/event_converter.c index ab48898d..8ed69412 100644 --- a/app/src/event_converter.c +++ b/app/src/event_converter.c @@ -96,6 +96,7 @@ convert_keycode(SDL_Keycode from, enum android_keycode *to, uint16_t mod, MAP(SDLK_RCTRL, AKEYCODE_CTRL_RIGHT); MAP(SDLK_LSHIFT, AKEYCODE_SHIFT_LEFT); MAP(SDLK_RSHIFT, AKEYCODE_SHIFT_RIGHT); + MAP(SDLK_LALT, AKEYCODE_ALT_LEFT); } if (!(mod & (KMOD_NUM | KMOD_SHIFT))) { @@ -120,7 +121,7 @@ convert_keycode(SDL_Keycode from, enum android_keycode *to, uint16_t mod, return false; } - if (mod & (KMOD_LALT | KMOD_RALT | KMOD_LGUI | KMOD_RGUI)) { + if (mod & (KMOD_RALT | KMOD_LGUI | KMOD_RGUI)) { return false; } // if ALT and META are not pressed, also handle letters and space diff --git a/server/src/main/java/com/genymobile/scrcpy/Device.java b/server/src/main/java/com/genymobile/scrcpy/Device.java index a8fdf677..8f8a7128 100644 --- a/server/src/main/java/com/genymobile/scrcpy/Device.java +++ b/server/src/main/java/com/genymobile/scrcpy/Device.java @@ -175,6 +175,7 @@ public final class Device { public boolean injectKeyEvent(int action, int keyCode, int repeat, int metaState) { long now = SystemClock.uptimeMillis(); + Ln.d("action=" + action + "; keycode=" + keyCode + "; metastate = " + metaState); KeyEvent event = new KeyEvent(now, now, action, keyCode, repeat, metaState, KeyCharacterMap.VIRTUAL_KEYBOARD, 0, 0, InputDevice.SOURCE_KEYBOARD); return injectEvent(event); ```