Open inferrna opened 1 year ago
Hi,
Good Android IME support is something that still needs work in Egui, Winit and android-activity unfortunately but is something I'd like to make some progress on.
You can see an initial attempt I made at enabling input method support in android-activity
here: https://github.com/rib/android-activity/pull/24 but I hit some odd input issues at the time which I didn't have a chance to debug.
More recently I also looked at creating initial branches for Winit + Egui to experiment with this too: https://github.com/rib/winit/tree/android-activity-ime-events https://github.com/rib/egui/tree/android-winit-ime-support
I would probably recommend first looking at the GameActivity
backend if you are looking to enable soft keyboard support since we should be able to utilize the GameText
bindings in that backend to also support the InputMethod protocol.
Maybe you could take a look over some of the above branches to see if they are a helpful starting point.
At least in terms of using JNI to access the Activity
you can use the ndk_context
crate to get to the Activity
, e.g. something like:
let ctx = ndk_context::android_context();
let jvm_ptr = ctx.vm();
let jvm = unsafe { jni::JavaVM::from_raw(jvm_ptr.cast()).expect("Expected to find JVM via ndk_context crate") };
let env = jvm.attach_current_thread_permanently().unwrap();
let activity_ptr = ctx.context();
let activity = unsafe { jni::objects::JObject::from_raw(activity_ptr as jni::sys::jobject) };
ndk_context
provides a general (also usable for native Rust libraries that don't know whether they are part of a standalone Rust application) way to access a JavaVM
on Android and an Activity
/Context
reference. android-activity
initializes the state for ndk_context
before your android_main()
runs.
There is an implement in miniquad/miniquad, use a normal Activity. https://github.com/not-fl3/macroquad/pull/526 https://github.com/not-fl3/miniquad/search?q=showKeyboard
@tkkcc if you just want to show the keyboard you should be able to use https://docs.rs/android-activity/0.4.0/android_activity/struct.AndroidApp.html#method.show_soft_input which should be equivalent to what macroquad does.
...but, unfortunately, that won't be very useful without also having input method support too.
macroquad has a custom subclass of the Activity
class which effectively lets you call the InputMethodManager::showSoftInput
(https://developer.android.com/reference/android/view/inputmethod/InputMethodManager#showSoftInput(android.view.View,%20int)) API via JNI
In android-activity
the show_input_method
API should do be equivalent to what macroquad does:
When android-activity is built for NativeActivity
It calls an NDK API called ANativeActivity_showSoftInput
which will end up calling an internal android_NativeActivity_setWindowFlags
(https://github.com/aosp-mirror/platform_frameworks_base/blob/ffeb11af1c22a1b8e2af57aac338e1fb57261ec5/core/jni/android_app_NativeActivity.cpp#L202) and using some IPC to invoke this: https://github.com/aosp-mirror/platform_frameworks_base/blob/ffeb11af1c22a1b8e2af57aac338e1fb57261ec5/core/jni/android_app_NativeActivity.cpp#L249 and that will call a showIme
method for NativeActivity.java
- similar to the macroquad subclass of Activity
: https://android.googlesource.com/platform/frameworks/base.git/+/master/core/java/android/app/NativeActivity.java#329
When android-activity is built for GameActivity
the show_soft_input
will call an internal GameActivity_showSoftInput
that uses IPC like in NativeActivity to send a CMD_SHOW_SOFT_INPUT
command that then calls GameTextInput_showIme
which then calls a JNI method called setSoftKeyboardActive
that also does the same thing as macroquad: https://android.googlesource.com/platform/frameworks/opt/gamesdk/+/refs/heads/master/game-text-input/src/main/java/com/google/androidgamesdk/gametextinput/InputConnection.java#113
I've at least smoke tested that show_soft_input
has worked for me in the past when working with experimental branches to start enabling better IME support so I'm not currently sure why nothing was happening for @inferrna when calling show_soft_input
via the linked example code.
I have a similar issue, I do use show_soft_input
(in a native activity) but nothing happens. I just get the following message:
InputMethodManager: Ignoring showSoftInput() as view=android.app.NativeActivity$NativeContentView{a4db4 V.ED..... ......ID 0,0-1080,2084} is not served.
agdk-eframe demo is able to show and hide input method via button
i don't know if clone an AndroidApp will cause something wrong. i don't know how to access options.event_loop_builder from ui update fn.
https://github.com/rust-mobile/android-activity/assets/17373509/bf4ab2b1-82e0-412b-a40e-463ae326343f
now i can achieve some level (very buggy) of input experience. based on agdk-eframe demo, picking the last commit of
and in game_activity/mod.rs
, i put the check of textInputState
before the check of android_app_swap_input_buffers
. because the second is null, when the first is valid.
pub fn input_events<F>(&self, mut callback: F)
where
F: FnMut(&InputEvent) -> InputStatus,
{
// eprintln!("555 {}", (*app_ptr).textInputState);
unsafe {
let app_ptr = self.native_app.as_ptr();
if (*app_ptr).textInputState != 0 {
let state = self.text_input_state();
callback(&InputEvent::TextEvent(state));
(*app_ptr).textInputState = 0;
}
}
let buf = unsafe {
let app_ptr = self.native_app.as_ptr();
let input_buffer = ffi::android_app_swap_input_buffers(app_ptr);
if input_buffer.is_null() {
return;
}
InputBuffer::from_ptr(NonNull::new_unchecked(input_buffer))
issues:
I'm also looking into keyboard support on android (and iOS). Based on @rib s work I have been able to get pretty decent support working on android 🥳
https://github.com/rust-mobile/android-activity/assets/8009393/5ac5299f-16d2-429e-899c-6d1e8d31987d
(opening and closing the keyboard also works, even if it's not in the video)
Unfortunately for this to work, I had to add completely new events to winit, basically passing the TextInputState struct from android activity all the way through to egui, and then handling the logic there. This is not ideal, but I'm not sure how much we could improve on this, with the way the GameTextInput works.
The most difficult part was finding the right places in egui to send the correct state updates when selecting a new text field or updating the selection.
If you'd like to try this you can use my relevant branches by adding this to your Cargo.toml (your project must be using game-activity for keyboard input to work):
[patch.crates-io]
winit = { git = "https://github.com/lucasmerlin/winit", branch = "v0.28.x_ime_support" }
egui = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"}
eframe = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"}
egui-wgpu = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"}
android-activity = { git = "https://github.com/lucasmerlin/android-activity", branch = "ime_support"}
These are the relevant branches: https://github.com/lucasmerlin/winit/tree/mobile_ime_support https://github.com/lucasmerlin/egui/tree/mobile_ime_support https://github.com/lucasmerlin/android-activity/tree/ime_support (based on the work of @rib)
For this to feel native on android what's still missing would be egui showing the composition range with underlined text instead of the blue selection highlight. But I think that should be pretty straightforward.
Also missing of course is to update the insets / resize the window when the keyboard opens.
Next, I'll look into iOS input support, I already have opening the keyboard and typing working but I'll also try to get autocomplete and suggestions.
show_soft_input() is still not working in NativeActivity with winit 0.29.10 (android-activity 0.5.0), nothing happens, no error on Android 8.1 or Android 11. I call it in android_main and after a touch event.
On Android 13 I get: InputMethodManager: Ignoring showSoftInput() as view=android.app.NativeActivity$NativeContentView{454c3ca V.ED..... ......ID 0,0-1080,2023} is not served.
I suspect this has to do with using a GPU surface. I will see if the window flags are correct (if I manage to access them).
https://github.com/rib/android-activity/pull/24 is a dead link.
GameActivity examples (rust-android-examples) immediately crash on my devices (Android 8.1, Android 11 and Android 13) with various errors, so I can't consider that for production. On Android 13 the error is undefined symbol: ANativeActivity_onCreate; this may have to do with cargo apk/xbuild, they're quite limited.
I'll try to do it through JNI.
https://github.com/rib/android-activity/pull/24 is a dead link.
Since this repository was migrated/transferred to the rust-mobile
organization (whew you created the issue), then Rib forked the repo again for personal contributions, GitHub no longer redirects. Instead, replace rib
with rust-mobile
to find the right link: https://github.com/rust-mobile/android-activity/issues/24
Ok, thanks. It looks like a View is needed (created in onCreate) to request focus first, as done here: https://android.googlesource.com/platform/frameworks/base/+/master/core/java/android/app/NativeActivity.java
Shouldn't ANativeActivity_onCreate simply call it's super.onCreate?
Shouldn't ANativeActivity_onCreate simply call it's super.onCreate?
Why would that be required, given that NativeActivity
exists to not have to write Java/JNI (to at least get a basic app going)?
The Java layer that powers NativeActivity
does call super.onCreate
:
It's unclear what line your link is supposed to link to.
Ok, I see. I managed to get GameActivity to work with soft input (but not the latest winit/wgpu versions) on Android 8.1.
I guess you don't get soft input on a window surface; you'd need a view.
It might be possible to achieve this with NativeActivity if you create a SurfaceView in the same way that is done for GameActivity.
But I guess that's kind of deprecated and people will just have to use GameActivity for soft input on a GPU surface.
It works with NativeActivity if you use getDecorView
, here's an example which worked for me
https://github.com/mvvvv/StereoKit-rust/blob/master/src/tools/os_api.rs#L196
Or it may be a few lines of Java, something like
public static void JNI_showKeyboard() {
InputMethodManager im = (InputMethodManager)Main.getContext().getSystemService(Context.INPUT_METHOD_SERVICE);
im.showSoftInput(window.getDecorView(), InputMethodManager.SHOW_IMPLICIT);
}
Still, there's a number of issues which prevent it to be usable in current state:
android:windowSoftInputMode="adjustResize"
property
Hi. I'm trying to display soft keyboard in simple app by calling
show_soft_input
: https://github.com/inferrna/hello_world_android_egui/blob/try_keyboard/src/lib.rs#L357 but nothing happens, even logcat contains no messages related to soft input, keyboard, etc.Googling I found a lot complains about ANativeActivity_showSoftInput() There is another way to do it — via JNI call. But it requires access to native_activity, which is currently hidden.