[Bug]: hardware keyboard does not work with IME

[knyipab]

Problem description

Not sure weather this counts as bug. However, hardware keyboard only types English characters and does not work with IME.

What steps will reproduce the bug?

Use any IME like Samsung keyboard with external keyboard to type.

What is the expected behavior?

I suppose users expect hardware keyboard to work with input methods.

A dumb approach might be reading the implementation in the source code of TextView in ASOP, but I don't find it easy.

[twaik]

What exactly do you mean? Works fine on my devices.

[knyipab]

In the below video, I typed using hardware keyabord and tried with different input methods (e.g. simplified chinese, japanese and traditional chinese) but it only output English. Do you have different behaviour on your device?

https://github.com/termux/termux-x11/assets/46513942/3f90831f-1701-47a8-bddf-83726c09f632

[twaik]

Toggle extra keys bar and swipe it to see the text input. Try to input something with additional keyboard there. You will see exactly the same behaviour. Only English symbols.

[knyipab]

Only with the help of EK bar text input, hardware keyboard + IME could work. They could not work directly with the LorieView. It is great and more natural to input into LorieView directly. I found RVNC Viewer seems to work well with hardware keyboard IME, perhaps they simply implement the TextView.

By the way, during my test, I found that Termux App could enable direct hardware keyboard IME after tapping the EK bar text input and then tapping into the terminal, for unknown reason. Shown in below video.

https://github.com/termux/termux-x11/assets/46513942/d43ff8bb-7d4e-4f44-8ed2-3905a5ddfda3

And this does not happen in Termux:X11. Shown in below video.

https://github.com/termux/termux-x11/assets/46513942/d7c96376-404a-4ce1-8c68-b0797953a15a

[twaik]

For some reason on my device it works using real hardware keyboard but does not work with scrcpy in OTG mode. So actually I can not reproduce it on my device.

[twaik]

And now it works with scrcpy too. Adorable.

[twaik]

Ok, I got it. It works with scrcpy in OTG mode, but does not work scrcpy with regular input. It seems like scrcpy injects events as a virtual keyboard device and Samsung's IME does not want to treat this device as capable to use multi-layout input.

[knyipab]

I have no clues about which part of code causing the bug because IME in hardware keyboard never work with Termux:X11, neither Gboard, for me since day 1. My device is Samsung Z Fold5 running One UI 6.1 Android 14.

Anything I can help debugging? Breakpoints? Logs? More test cases (I own a S21)?

[twaik]

I think we can try to get symbols from IME using InputConnection, but that may break regular input mechanisms of X11 programs. I am not sure if they will work at all.

[knyipab]

Just tested and I don't think so.

I applied breakpoints on any methods of BaseInputConnection in #620 . It does not trigger any InputConnection methods when typing in hardware keyboard. I also tested on my S21 and it's just the same.

[twaik]

It should trigger commitText.

[knyipab]

No, I can confirm that hardware keyboard on my devices does not trigger commitText at all. I tried tapping software keyboard, and that tapping triggers commitText.

[knyipab]

Same no matter it is BT or USB (OTG) keyboard.

[twaik]

Try to remove onKeyPreIme method.

[knyipab]

Where and how to? Searched and didn't find that method.

Tried @Override that method and return false for LorieView but still it doesn't trigger commitText, neither any methods of InputConnection.

[twaik]

Sorry, forgot the exact name. https://github.com/termux/termux-x11/blob/ee1bd84fabf19d59302ca11c12d575e14e044be1/app/src/main/java/com/termux/x11/LorieView.java#L205-L209

[knyipab]

Yes. By removing this method, the IME works perfectly! And tapping soft keyboard continues to work, not affected.

[knyipab]

Any reason to call mLorieKeyListener.onKey instead of super.dispatchKeyEventPreIme? They do similar thing in my test except that the latter can also handle IME for hardware keypresses.

[twaik]

I am not sure why but there some devices (or IMEs) which modify key event data. I can make it optional if it causes problems.

[knyipab]

Yes, please. That will be very helpful.

[knyipab]

https://github.com/termux/termux-x11/commit/3c8cc7591dc7d0a8c6f21afed915c42e1a980b48 does not work without setting outAttrs.inputType.

My previous test results are based on this PR #620 (cuz you mentioned BaseInputConnection at the beginning).

If you don't want to merge PR #620 any time soon, may consider to add this to LorieView.

    @Override
    public InputConnection onCreateInputConnection(EditorInfo outAttrs) {
        outAttrs.inputType = InputType.TYPE_CLASS_TEXT | InputType.TYPE_TEXT_FLAG_NO_SUGGESTIONS | InputType.TYPE_TEXT_VARIATION_VISIBLE_PASSWORD;
        return super.onCreateInputConnection(outAttrs);
    }