Support `ReceivedCharacter` on Android

[Hoodad]

Hello!

On Android the ReceivedCharacter event is not implemented. This could be due to the fact that there does not appear to be any way to get the unicode character using NDK as it stands today. So any key pressed using the virtual keyboard will have to rely on the keycode and metadata state which limits the number of characters supported to A-Z, 0-9 and some special characters. The issues was mentioned in this thread but I feelt this deserves its own issue. Maybe its something I missed, and it is possible to get the unicode character?

I created a fork that does a lot of the work for our (Embark) use-case for now, but its a bit hacky and limited in its functionality there for not something that I feel should be merged to winit.

[rib]

Just in case there might be an XY problem here I was wondering - are you more just looking to have decent support for text input on Android, which should support full unicode input (e.g. via https://developer.android.com/games/agdk/add-support-for-text-input which was mentioned in the linked thread) or maybe more specifically want ReceivedCharacter support? Or both :-p

Something related that I noticed recently was that in Java the KeyEvent API does actually have a getUnicodeChar() method which looks like it could help with supporting ReceivedCharacter events. Although that's not exposed via the NDK input API it's also something that should be easy to expose via the game-activity glue layer I experimented with (since that's built directly on the Java input API and doesn't go via the NDK input wrapper API.

[Hoodad]

Yeah my wording is not 100% as I'm new in the space. The end goal would be to have the virtual keyboard character a user presses to be sent as ReceivedCharacter from winit like it does on other platforms (correct me if this is wrong?). This functionality requires full unicode input support, so I think I'm asking for both unless there is a different way that I'm missing 😅

[rib]

I think the main difference I'd expect is that for text input on Android it's usually via a virtual keyboard and so then I guess the trickier part is dealing with adjusting your game/UI to fit any current text box in the visible space above the keyboard and then responding to higher-level input method events. That makes text input handling more like handling IMEs on other platforms, instead of responding to lower-level keyevents - which mainly make sense with physical input devices.

E.g. quoting from the KeyEvent docs:

"As soft input methods can use multiple and inventive ways of inputting text, there is no guarantee that any key press on a soft keyboard will generate a key event: this is left to the IME's discretion, and in fact sending such events is discouraged. You should never rely on receiving KeyEvents for any key on a soft input method. In particular, the default software keyboard will never send any key event to any application targetting Jelly Bean or later, and will only send events for some presses of the delete and return keys to applications targetting Ice Cream Sandwich or earlier. Be aware that other software input methods may never send key events regardless of the version."

So I suppose I was curious to try and gauge a bit more about your higher-level use case - e.g. if you care more about physical keyboards + input devices then I can see how that could be practically supported via KeyEvent's getUnicodeChar() but if you're thinking more about virtual keyboard input then I'm not so sure you're really interested in KeyEvents or ReceivedCharacter events.

It might make sense that Winit should also support a higher-level portable IME API that caters to soft keyboard input on Android (hopefully supporting all the features of https://developer.android.com/games/agdk/add-support-for-text-input) . I think there has been some work on IME support in general for desktop window systems but I'm not sure atm how well that maps to Android.

For reference: https://github.com/rust-windowing/winit/issues/1823 https://github.com/rust-windowing/winit/pull/2243

Comparable wayland protocol: https://wayland.app/protocols/text-input-unstable-v3#zwp_text_input_v3

[kchibisov]

It might make sense that Winit should also support a higher-level portable IME API that caters to soft keyboard input on Android (hopefully supporting all the features of https://developer.android.com/games/agdk/add-support-for-text-input) . I think there has been some work on IME support in general for desktop window systems but I'm not sure atm how well that maps to Android.

The IME api was merged and text-input-v3 is implemented on Wayland. Basically you have Preedit(text, cursor_position) and Commit(String). I'd assume that could map to android or you'd need more of that?

[rib]

Taking a bit more of a look at this; the two main things I'm uncertain about currently are:

1. tracking a user selection that's separate from the compose region, which Android IMEs expect to be able to track (atm, conceptually Winit tracks a single compose region)
2. helping applications know when they should be controlling the IME position via set_ime_position vs when they should instead be updating their UI to keep an input region visible while a soft keyboard is visible

I've added a more detailed comment here: https://github.com/rust-windowing/winit/issues/1497

[kchibisov]

I have no idea wrt on-screen-keyboards on android and briefly familiar with Wayland onces, and those don't require any changes in winit at all to work from what I know right now. Though, that might change in the future.

[rib]

Yeah, simple onscreen keyboards handled in a Wayland compositor can probably just be implemented to look like regular keyboard input (i.e. like a physical keyboard), transparent to the application since input is proxied via the compositor.

Android on screen keyboards are more comparable to desktop IMEs except that you don't use set_ime_position with them. (the IME UI is the fixed on-screen keyboard)

Android has a separate mechanism for reporting changes to 'insets' that let the app know when an onscreen keyboard is visible, which isn't currently a concept that exists in Winit.

I'm guessing that the current interface can be used to bootstrap some basic IME/on-screen-keyboard support on Android by basically reporting that there is never any selection - and hopefully most IMEs will handle that gracefully, I'm not really sure atm.

Other details will be with deciding how to request for the IME/on-screen-keyboard to show and hide. Android has a few different states to account for users being able to forcibly enable an IME UI - I think for accessibility needs - and so there's some distinction for applications to be able to say "really hide the IME" vs "hide the IME if the user hasn't forcibly shown an IME".

[Hoodad]

So I suppose I was curious to try and gauge a bit more about your higher-level use case - e.g. if you care more about physical keyboards + input devices then I can see how that could be practically supported via KeyEvent's getUnicodeChar() but if you're thinking more about virtual keyboard input then I'm not so sure you're really interested in KeyEvents or ReceivedCharacter events.

The sole reason I asked for the ReceivedCharacter event was just to have functionality parity with desktop. Assuming that it would make egui and imgui just work on Android the same way it does on desktop builds. (Which they did even once I introduced the events in above mentioned fork, though current implementation is heavily limited. At least for now it suffices).

While GameActivity has switched to a polling from an event driven design this means that there may be a considerable amount of time until the app fetches data again. The implication being that users could have typed multiple characters since last poll, so to not loss characters I think the Android API added ability to receive "text" or "Strings" rather then just an individual "char". While this could have been achieved by queueing up a list of character events they know the app will merge into a text they have kindly merged individual chars to create a text so next poll will get the last text input since previous poll. But that's just a guess. I'm not sure if winit today has any support for receiving a text rather then individual characters, if it does that would be great as I don't see any particular reason why individual characters would be preferred or required. If a new event is needed I would consider having a fallback from Android to take a text and split it into individual characters just to make sure backwards compatibility is there for already existing API's and integrations.

Regarding the IME position, not sure if that is part of this issue. Handling of the UI and placement of the IME is in my view a separate issue.

[rib]

The sole reason I asked for the ReceivedCharacter event was just to have functionality parity with desktop.

yip, makes sense

While GameActivity has switched to a polling from an event driven design this means that there may be a considerable amount of time until the app fetches data again.

To clarify a bit; GameActivity batches input events and apps are generally expected to pull them in sync with their rendering updates. This is more about avoiding lots of redundant mainloop wake ups and assuming that games will be waking up regularly to redraw. (so more of a game-centric optimization that shouldn't really affect input handling in terms of text vs characters)

I'm not sure if winit today has any support for receiving a text rather then individual characters, if it does that would be great

Right this is what the Winit "IME" API is basically about. The terminology is borrowed from how desktops traditionally support text input for languages like Chinese and Japanese where you typically compose characters with an overlayed UI showing language-specific completions. On Android though an on-screen-keyboard is also an input method in much the same way as a desktop input method. These input methods work by intercepting the physical inputs and send you strings of text instead of characters and they also track a special "compose" region that is like a tentative/partial completion for whatever the user is currently typing.

The stuff about IME position is just because I was starting to think about how to support the Winit IME abstraction on Android to support on-screen-keyboards properly but one of the notable differences between the IME abstraction on Android and what you typically have for desktop window systems is that on desktops the application is responsible for telling the IME where in their window input is happening (e.g. the position of the focused text entry) so that it can place it's UI for showing character completions next to the text entry. On Android though the IME UI is the on-screen-keyboard which is in a fixed location.

[Hoodad]

Right this is what the Winit "IME" API is basically about. The terminology is borrowed from how desktops traditionally support text input for languages like Chinese and Japanese where you typically compose characters with an overlayed UI showing language-specific completions. On Android though an on-screen-keyboard is also an input method in much the same way as a desktop input method. These input methods work by intercepting the physical inputs and send you strings of text instead of characters and they also track a special "compose" region that is like a tentative/partial completion for whatever the user is currently typing.

Thanks for clarifying, I will have a look at the current IME efforts!

The stuff about IME position is just because I was starting to think about how to support the Winit IME abstraction on Android to support on-screen-keyboards properly but one of the notable differences between the IME abstraction on Android and what you typically have for desktop window systems is that on desktops the application is responsible for telling the IME where in their window input is happening (e.g. the position of the focused text entry) so that it can place it's UI for showing character completions next to the text entry. On Android though the IME UI is the on-screen-keyboard which is in a fixed location.

This is what had me a bit confused since I didn't consider that IME UI position is not always determined by the "operating system", like on Android. Sounds like something that is needed in the future, at least for completeness.

Though I feel that it deserves its own specific issue. Maybe create one about adding support for placement of IME UI?