Sending keyboard events as physical keyboard instead of virtual keyboard

[AlynxZhou]

Current it send keyboard events as virtual keyboard (like input method keyboard), however google pinyin input supports physical keyboard with OTG to input, physical can use numbers to choose words which is really convenient and saves half of screen, if it can send as physical keyboard it will be better (because actually we are using a physical keyboard on computer).

[rom1v]

See https://github.com/Genymobile/scrcpy/issues/87#issuecomment-378775565 about OTG.

[lasek101]

Hello First thanks for Your work. Now to my problem. I use polish programmer phisical keyboard. It is normal qwerty keyboard with polish national characters obtaining by pressing right alt with specific letter. I have compiled, on linux box, client and server fom dev and rawalpha branch but can't get any polish diacritical letter. Pressing ctrl+k does nothing. What I'm doing wrong?

[rom1v]

@lasek101 Unfortunately, I have no solution to generate such keys on the device. See https://github.com/Genymobile/scrcpy/issues/193 and https://github.com/Genymobile/scrcpy/pull/194.

[AlynxZhou]

@rom1v what about letting computer dealing with input method and just pass input string to phone?

[npes87184]

Hi, @AlynxZhou

Does the problem become to be able to input (Chinese) words from computer? If the answer is yes, it may be a simple workaround to cover the problem which is some character cannot be prefectly handled. However, there are still some problems to deal. For example, pass input string to where? clipboard or pass directly to the edittext?

If we pass the string to clipboard, there will be inconvenience to use. We need to paste again and again in phone. On the other hand, if we want to directly pass to input field. As I know, there are many restricts. For instance, how about webview?

In my opinion, unless we can cover these problems. It is not an elegant solution yet.

Thanks.

[lasek101]

Maybe there is a hope. If I connect physical keyboard in OTG mode I can choose layout in Android (settings-languages and input methods - physical keyboard - then I choose keyboard I've connected and choose correct layout). After that I can get polish diacritical letters by pressing right alt and certain letter. Mayby there is a way to tell android that keyboard used by scrcpy is like OTG one.

[rom1v]

@AlynxZhou

what about letting computer dealing with input method and just pass input string to phone?

This is what the client sends to the server (except for letters and space since v1.4, see #87).

The problem is that the device does not support injecting characters other than ASCII. See handle text inputs.

@lasek101

If I connect physical keyboard in OTG mode

See https://github.com/Genymobile/scrcpy/issues/87#issuecomment-378775565.

After that I can get polish diacritical letters by pressing right alt and certain letter.

When you press AltGr+letter on your OTG keyboard, or on your computer?

[lasek101]

I press AltGr+letter on phisical keyboard and get correct diacritical letter on android. see https://youtu.be/QEmYMImkNnQ

[rom1v]

@lasek101 OK, so this comment totally applies: https://github.com/Genymobile/scrcpy/issues/87#issuecomment-378775565

This works because an OTG keyboard sends HID events over USB directly, which is not possible from a computer (except using HID over AOA).

[lasek101]

So, this is definitely not possible in scrcpy?

[lasek101]

Alternativly mayby it is possible to use phisical keyboard like virtual in a way that: when I long press letter 'a' I get option to choose what character I want to send i.e. 'ą'. Now when I long press letter I get many reps of that letter.

[AlynxZhou]

@rom1v I think HID events is the best way however I know little about Android development and USB HID, if I know I can help...

[rom1v]

@lasek101

Alternativly mayby it is possible to use phisical keyboard like virtual in a way that: when I long press letter 'a' I get option to choose what character I want to send i.e. 'ą'.

The problem is not how to generate the key, but how to inject it without root access into the system. See Handle accented characters.

@AlynxZhou

I think HID events is the best way

It would be great, but it may be a bit complicated.

Since you don't necessarily use a USB keyboard on your computer, you need to retrieve the key events via SDL, and "convert" them into raw USB HID events (is the mapping easy?).

And you need to enable AOA (which does not work on all devices, and I don't know how to communicate over AOA from Windows). See discussion about audio forwarding: https://github.com/Genymobile/scrcpy/issues/14.

if I know I can help...

Generating HID events from SDL key events would be great :wink:

[AlynxZhou]

I understand what we need, but I am questioning about whether we have a proper way to simulate a "virtual device" on Android without root... It seems AOA is not proper way for this? Plus did you see KDE Connect remote keyboard? This seems a way that send text from PC as an input method, I am trying this, but unfortunately GSConnect cannot work after GNOME 3.30 upgrade...

[rom1v]

@AlynxZhou Yes, this is another alternative. It would require to install an APK though, so this may be "optional" (install the additional keyboard if you want to support "special characters"). See https://github.com/Genymobile/scrcpy/issues/37#issuecomment-372206555.

[AlynxZhou]

Installed KDE Plasma and test failed: KDE Connect cannot forward Chinese character too...

[amosbird]

Since you don't necessarily use a USB keyboard on your computer, you need to retrieve the key events via SDL, and "convert" them into raw USB HID events (is the mapping easy?).

Is there a PoC for this?

[rom1v]

@amosbird No.

[amosbird]

Ok. I asked that after coming across this https://github.com/rom1v/aoa-hid-bug . Looking forward for a PoC :D

[amosbird]

Finished a dirty PoC. It works as expected. https://github.com/amosbird/scrcpy/blob/master/app/src/usb_hid_keys.h#L482

[rom1v]

Cool, thank you for the PoC :+1:

I had to add the thread dependency to make it work:

diff --git a/app/meson.build b/app/meson.build
index 319548a..8b2456c 100644
--- a/app/meson.build
+++ b/app/meson.build
@@ -29,6 +29,7 @@ if not get_option('crossbuild_windows')
         dependency('libavutil'),
         dependency('sdl2'),
         dependency('libusb-1.0'),
+        dependency('threads'),
     ]

 else

(and of course change my pid:vid)

Currently, it does not handle special chars (e.g. if I press the key è, it sends 7), and inputs are interleaved with normal text input events (so it's hard to tell which method sends what).

Could you link some HID doc/spec you used to generate these tables, please? https://github.com/amosbird/scrcpy/blob/master/app/src/usb_hid_keys.h#L27

Do you know how to use USB on Windows with libusb (and make AOA work on Windows, too)?

[amosbird]

Currently, it does not handle special chars (e.g. if I press the key è, it sends 7), and inputs are interleaved with normal text input events (so it's hard to tell which method sends what).

yeah, that commit is broken. Now it works fine.

Could you link some HID doc/spec you used to generate these tables, please?

https://gist.github.com/MightyPork/6da26e382a7ad91b5496ee55fdc73db2

Do you know how to use USB on Windows with libusb (and make AOA work on Windows, too)?

Sorry I have no idea.

[amosbird]

@rom1v hmm, the media control is very unreliable via AKEYCODE_MEDIA_PLAY_PAUSE. How can I correctly fake the control signals generated by my bluetooth earphone, which works in all music applications?

[rom1v]

https://gist.github.com/MightyPork/6da26e382a7ad91b5496ee55fdc73db2

Thanks. The AOSP doc is there: https://source.android.com/devices/input/keyboard-devices

yeah, that commit is broken. Now it works fine.

Cool, text input are not interleaved anymore. However, I still can't press è (on the 7 key on AZERTY keyboard).

[amosbird]

I still can't press è

hmm, I'm not sure where that issue resides. Can you input è on a real external usb keyboard via otg?

[amosbird]

I've iterated all the combinations of the 4 bytes HID data. None of them play/pause the music for me....

[amosbird]

finally achieved media control. The HID interface is really dark.

[wuyuanyi135]

+1 to this feature. Would be fantastic to automatically solve input method problems.

[aeroxy]

How do I enable the raw_key_events? It doesn't seem to be in my scrcpy:

scrcpy 1.12.1

dependencies:

• SDL 2.0.10
• libavcodec 58.54.100
• libavformat 58.29.100
• libavutil 56.31.100

[rom1v]

@aeroxy No, it's not implemented, it was just an experimentation on a branch. Why do you need this?

[AlynxZhou]

@lasek101 OK, so this comment totally applies: #87 (comment)

This works because an OTG keyboard sends HID events over USB directly, which is not possible from a computer (except using HID over AOA).

@rom1v OK, 3 years past and I finally got enough experience to understand the code and the protocol! Now I bring some good news that I just finished re-writing of @amosbird 's code and tested on my Galaxy S9+, it works fine, Android starts to treat scrcpy as a physical keyboard and IME starts to work.

There are some mistakes in https://github.com/amosbird/scrcpy/commit/fcb36fbf870a220111b59b0fe55c27fbdec33e01, most about how HID keyboards work and I've read the official documents to properly convert SDL's keyboard events to HID events. And thanks to his code gives me basic idea of how to implement it.

Currently the code is dirty: I started my work on master branch instead of dev, some error handling are missing, some code has conflict with the old inject mode, and I am still not sure media key should be handled by keyboard or consumer, but I think I can fix those soon and send a pull request as soon as I can.

[AlynxZhou]

Ported to dev and added error handling, it will automatically fallback to inject mode, looks fantasy!

Let me take some time for the media keys. Also the mod handled by scrcpy...currently I just pass all scancodes I could to Android because this sounds more like a physical keyboard.

[AlynxZhou]

https://github.com/AlynxZhou/scrcpy/commit/62f8b1977996f9484d7674abf069be6fb480bcb5

My first commit is here and I'll try to add more commit for it.

[rom1v]

@AlynxZhou Cool :+1:

Just for info, did you also manage to make it work on Windows/macOS?

It's secondary and just for information: to avoid --usb, we could also select the USB device by serial (cf for ex https://github.com/rom1v/usbaudio/blob/6f46c43b9a04249f5de61255b834dc7ebd56421c/src/aoa.c#L141-L156). And we could call adb get-serialno to get the serial of the connected device, so the selection would be automatic. (But this will be for a second step, keep it simple for now :+1:)

[AlynxZhou]

@AlynxZhou Cool 👍

Just for info, did you also manage to make it work on Windows/macOS?

Not yet, I'd like to try to port it to windows after all features done but I don't have a mac. And it is especially useful for Chinese linux users because there is no QQ or WeChat for linux. XD

It's secondary and just for information: to avoid --usb, we could also select the USB device by serial (cf for ex https://github.com/rom1v/usbaudio/blob/6f46c43b9a04249f5de61255b834dc7ebd56421c/src/aoa.c#L141-L156). And we could call adb get-serialno to get the serial of the connected device, so the selection would be automatic. (But this will be for a second step, keep it simple for now 👍)

Really good, will try.

[AlynxZhou]

I still have some problems about media keys:

1. My keyboard do support consumer page which is used for media control, but if I press for example volume up, it will be caught by my desktop, so I think it might be not useful to implement media keys for scrcpy.
2. There are different ways to trigger media control, besides send media keys events via HID, I see current scrcpy's code uses send_keycode() to inject some android media keys, I'm not sure which one is better (but I am afraid of inject, maybe HID event is the correct way). Plus, I found that most composed keys do work natively on Android, for example if I press Ctrl+X on HID mode, it can cut text and save it to Android's clipboard, and Esc just works as back.
3. So, if I choose any one of the two choices, then I have another question about how to trigger those functions in HID mode, some actions like back/cut/paste works natively and I think it's better not to add scrcpy's MOD+X composed keys for them. But for media controls like volume up/next/pause (and maybe rotate screen, they are useful), those keys are blocked by desktop environment, is it ok to add MOD+X keys for them?

I think that if I "do the correct things", then I cannot keep consist between inject mode and HID mode.

[AlynxZhou]

Another advice: About https://github.com/Genymobile/scrcpy/blob/master/app/src/input_manager.c#L496-L506, I think it is wired because MOD+x means cut to Android clipboard, MOD+c means copy to Android clipboard, but MOD+v is paste from PC to Android! Then what about paste from Android clipboard? I personally think we should use MOD+v for paste from Android clipboard and some other keys, for example MOD+Insert for paste from PC to Android.

[rom1v]

because MOD+x means cut to Android clipboard, MOD+c means copy to Android clipboard

AND to computer clipboard.

So if you MOD+x or MOD+c on Android, you can paste on your computer. Conversely, if you Ctrl+c from somewhere on your computer, you can MOD+v to paste to Android.

As a particular case, if you MOD+c then MOD+v from scrcpy on the device, then it still works (it just incurs an additional synchronization from/to the computer, but the result is as expected).

[rom1v]

Plus, I found that most composed keys do work natively on Android, for example if I press Ctrl+X on HID mode, it can cut text and save it to Android's clipboard, and Esc just works as back.

MOD(s) key(s) must never be forwarded to the device. Also, by default, MOD is Alt, especially to let Ctrl forwarded to the device.

See this if (smod): https://github.com/Genymobile/scrcpy/blob/c96f5c70e9c6fe82da059c7d24e77d9730aa7f86/app/src/input_manager.c#L427 (and the return; at the end of the block).

[AlynxZhou]

Plus, I found that most composed keys do work natively on Android, for example if I press Ctrl+X on HID mode, it can cut text and save it to Android's clipboard, and Esc just works as back.

MOD(s) key(s) must never be forwarded to the device. Also, by default, MOD is Alt, especially to let Ctrl forwarded to the device.

See this if (smod):

https://github.com/Genymobile/scrcpy/blob/c96f5c70e9c6fe82da059c7d24e77d9730aa7f86/app/src/input_manager.c#L427 (and the return; at the end of the block).

Yes, I know smod is not forwarded to Android, but I am considering not to do that for HID mode, to behavior more like a real keyboard. (But maybe I'll use it to simulate media keys that caught by DE) XD

[AlynxZhou]

because MOD+x means cut to Android clipboard, MOD+c means copy to Android clipboard

AND to computer clipboard.

So if you MOD+x or MOD+c on Android, you can paste on your computer. Conversely, if you Ctrl+c from somewhere on your computer, you can MOD+v to paste to Android.

As a particular case, if you MOD+c then MOD+v from scrcpy on the device, then it still works (it just incurs an additional synchronization from/to the computer, but the result is as expected).

WOW, I just tried and it's really cool, except when I press MOD+v, it pastes my Android clipboard's content out and then paste computer's content inside Android's clipboard (maybe it should not make android clipboard's content out).

Maybe I'll add this to HID mode too XD

[rom1v]

except when I press MOD+v, it pastes my Android clipboard's content out and then paste computer's content inside Android's clipboard (maybe it should not make android clipboard's content out).

I'm not sure to understand.

Maybe I'll add this to HID mode too XD

IMO, there is nothing specific to HID here, the MOD key should never be forwarded, so that the user can still execute scrcpy shortcuts. Injection or HID does not change anything in this regard.

[AlynxZhou]

except when I press MOD+v, it pastes my Android clipboard's content out and then paste computer's content inside Android's clipboard (maybe it should not make android clipboard's content out).

I'm not sure to understand.

I cut aaaaa on android, then cut bbbbb on computer, then go to scrcpy, press MOD+v, aaaaa goes in to text field, bbbbb shows on my gboard (which means it's in clipboard).

What I expected: bbbbb shows on my gboard, no aaaaa.

Anyway this is not important.

Maybe I'll add this to HID mode too XD

IMO, there is nothing specific to HID here, the MOD key should never be forwarded, so that the user can still execute scrcpy shortcuts. Injection or HID does not change anything in this regard.

Gboard uses HID keyboard's alt key for emoji selector, but I think pressing alt only is not affected by MOD key.

I'll try to implement volume up/down with media keys (user press MOD+up, scrcpy sends HID_MEDIA_VOLUME_UP instead of using action_volume_up()), and other use MOD just like things already exists.

[rom1v]

I cut aaaaa on android, then cut bbbbb on computer, then go to scrcpy, press MOD+v, aaaaa goes in to text field, bbbbb shows on my gboard (which means it's in clipboard).

If I do the same on current master, bbbbb goes into the tex field, and bbbbb is in the Android clipboard (aaaaa is totally overwritten).

Gboard uses HID keyboard's alt key for emoji selector, but I think pressing alt only is not affected by MOD key.

So that would be very annoying: if I press Alt+f to make scrcpy fullscreen, it would open the emoji selector because I pressed Alt.

See discussions #1598 #1465 #1446. For Alt, especially: https://github.com/Genymobile/scrcpy/pull/1465#issuecomment-637168718

[AlynxZhou]

I cut aaaaa on android, then cut bbbbb on computer, then go to scrcpy, press MOD+v, aaaaa goes in to text field, bbbbb shows on my gboard (which means it's in clipboard).

I'm on dev but maybe not the latest commit.

If I do the same on current master, bbbbb goes into the tex field, and bbbbb is in the Android clipboard (aaaaa is totally overwritten).

Gboard uses HID keyboard's alt key for emoji selector, but I think pressing alt only is not affected by MOD key.

So that would be very annoying: if I press Alt+f to make scrcpy fullscreen, it would open the emoji selector because I pressed Alt.

See discussions #1598 #1465 #1446. For Alt, especially: #1465 (comment)

Not a problem, only Alt triggers Gboard's selector, Alt+f won't. I'll do further test for best experience.

[rom1v]

I'll try to implement volume up/down with media keys (user press MOD+up, scrcpy sends HID_MEDIA_VOLUME_UP instead of using action_volume_up()), and other use MOD just like things already exists.

IMO, there should be an abstraction over the injection method.

Pressing MOD+↑ should call action_volume_up(), which should inject a "volume up" event to the device, regardless of the method (InputManager or HID). Same for Alt and everything independent of the injection method.

[AlynxZhou]

I cut aaaaa on android, then cut bbbbb on computer, then go to scrcpy, press MOD+v, aaaaa goes in to text field, bbbbb shows on my gboard (which means it's in clipboard).

I'm on dev but maybe not the latest commit.

If I do the same on current master, bbbbb goes into the tex field, and bbbbb is in the Android clipboard (aaaaa is totally overwritten).

Gboard uses HID keyboard's alt key for emoji selector, but I think pressing alt only is not affected by MOD key.

So that would be very annoying: if I press Alt+f to make scrcpy fullscreen, it would open the emoji selector because I pressed Alt. See discussions #1598 #1465 #1446. For Alt, especially: #1465 (comment)

Not a problem, only Alt triggers Gboard's selector, Alt+f won't. I'll do further test for best experience.

I'm wrong, I got the alt problem too, but I have an idea that we can use some composed key for raw alt in HID mode.

[AlynxZhou]

I'll try to implement volume up/down with media keys (user press MOD+up, scrcpy sends HID_MEDIA_VOLUME_UP instead of using action_volume_up()), and other use MOD just like things already exists.

IMO, there should be an abstraction over the injection method.

Pressing MOD+↑ should call action_volume_up(), which should inject a "volume up" event to the device, regardless of the method (InputManager or HID). Same for Alt and everything independent of the injection method.

I have the following code:

            case SDLK_DOWN:
                if (control && !shift) {
                    // forward repeated events
                    // Press volume down.
                    unsigned char buffer[HID_MEDIA_KEY_LENGTH] = {
                        HID_MEDIA_REPORT_ID,
                        HID_MEDIA_KEY_VOLUME_DOWN
                    };
                    aoa_send_hid_event(im->usb_handle, buffer,
                        HID_MEDIA_KEY_LENGTH);
                    // Release volume down.
                    buffer[1] = HID_MEDIA_KEY_UNDEFINED;
                    aoa_send_hid_event(im->usb_handle, buffer,
                        HID_MEDIA_KEY_LENGTH);
                }
                return;

This has the same function as action_volume_up() but not injected, it's HID event. How about this?

[AlynxZhou]

MOD+m is used for menu, I am not sure what for Mute now (laughing)

[AlynxZhou]

https://github.com/AlynxZhou/scrcpy/commit/18bda29c6540ac32d3042fc635ff2e357846b2e1

This commit just add MOD keys support to HID mode, I just reuse existed code, not modify it. (Except Ctrl+v, it is a native shortcut for Android, I suggest not to alternate it?)