search

rust-windowing / winit

Window handling library in pure Rust
https://docs.rs/winit/
Apache License 2.0
4.74k stars 890 forks source link

Tracking issue for IME / composition support #1497

Open murarth opened 4 years ago

murarth commented 4 years ago

This issue tracks the implementation of IME composition events on each platform, an API initially proposed in #1293. PRs implementing this event should be made against the composition-event branch, which will be merged into master once all implementations are complete.

lucasmerlin commented 1 year ago

I managed to add IME support to android and iOS: Android input works really well, including autocomplete and suggestions:

Video

https://user-images.githubusercontent.com/8009393/251972889-5ac5299f-16d2-429e-899c-6d1e8d31987d.mp4

iOS supports basic text entry, but no autocomplete /autocorrect yet:

Video

https://github.com/rust-windowing/winit/assets/8009393/f1836c34-8131-4578-915a-0a0dfbd11ca8

The android support is based on @rib's work on android-activity here: https://github.com/rust-mobile/android-activity/pull/24 I had to add a new Event to winit, basically just passing through the TextInputState to egui, handling the logic there. Implementation in egui is pretty simple, basically

The TextInputState struct looks like this:

Expand me

```rust /// This struct holds a span within a region of text from `start` (inclusive) to /// `end` (exclusive). /// /// An empty span or cursor position is specified with `Some(start) == Some(end)`. /// /// An undefined span is specified with start = end = `None`. #[derive(Debug, Clone, Copy, PartialEq, Eq)] #[cfg_attr(feature = "serde", derive(Serialize, Deserialize))] pub struct TextSpan { /// The start of the span (inclusive) pub start: Option, /// The end of the span (exclusive) pub end: Option, } #[derive(Debug, Clone, PartialEq, Eq)] #[cfg_attr(feature = "serde", derive(Serialize, Deserialize))] pub struct TextInputState { pub text: String, /// A selection defined on the text. pub selection: TextSpan, /// A composing region defined on the text. pub compose_region: TextSpan, } ```

For iOS text support, I implemented the UIKeyInput api to receive basic key events but it doesn't support autocomplete / autocorrect / character composition yet. There also is UITextInput to support these features, I think it should be possible to write a implementation in winit that provides the same simple TextInputState api to applications, but I don't have the objective c skills to implement this.

If you want to try these in an egui app, you can add the following to your Cargo.toml to try my branches:

[patch.crates-io]
winit = { git = "https://github.com/lucasmerlin/winit", branch = "v0.28.x_ime_support" }
egui = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"}
eframe = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"}
egui-wgpu = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"}
android-activity = { git = "https://github.com/lucasmerlin/android-activity", branch = "ime_support"}

Or try this example for android: https://github.com/lucasmerlin/rust-android-examples/tree/ime_support_showcase/agdk-eframe.

These are the relevant branches: https://github.com/lucasmerlin/winit/tree/mobile_ime_support https://github.com/lucasmerlin/egui/tree/mobile_ime_support https://github.com/lucasmerlin/android-activity/tree/ime_support (based on the work of @rib)

If there is interest to add this to winit I'm happy to open a draft PR.

kchibisov commented 1 year ago

Well, we'd need patches upstream to be merged first, you could open a draft PR if you want to though.

rib commented 1 year ago

Sorry for the delay following up here @lucasmerlin - very cool that you got something working here.

I'm hoping to get a chance to look at this soon.

It'll be good to compare the egui / winit changes with the ones I experimented with at the time:

https://github.com/rib/winit/tree/android-activity-ime-events https://github.com/rib/egui/tree/android-winit-ime-support

It'll be good to test this on GameActivity 2.0.2 once I land this PR: https://github.com/rust-mobile/android-activity/pull/88

phnaharris commented 6 months ago

So first of all I don't need CompostionDone event, since this is an event when I'd send things downstream, meaning that I don't need CompositionDeleteSurroundingText. And it seems like I don't need DeleteSurroundingText, since I can model it via CompositionPreedit. CompositionCommit(text: String) is indeed needed so the user will know when to insert text.

Hi @kchibisov, I'm wondering why you said that you don't need CompositionDeleteSurroundingText event and how we can model it via CompositionPreedit. Because, the surrounding text is the text around the cursor, even if user committed it to widget or not, as far as I know. I think what you mean is we can modify the preedit, but the problem became a thing if the IME want to delete the surrounding text via text input v3 after the preedit be committed to widget, I think we didn't have a mechanism for doing that yet.

Now I wanna support this feature about surrounding text. Based on the design of GTK for this feature, I need a way for downstream (I'm using egui) to send the surrounding text back to winit whenever it need, through Ime::RetrieveSurrounding event. It would be nice if you have any idea for doing that.

kchibisov commented 6 months ago

You should add delete surrounding text, just keep in mind that it doesn't exist without setting surrounding text. The issue is how to communicate that cross platform, but it's a technical detail.

Part of that was done in https://github.com/rust-windowing/winit/pull/2993 . IME will also be reworked for 0.31.

phnaharris commented 6 months ago

it doesn't exist without setting surrounding text

I've taken the work in this pull request and got the idea that how the surrounding text be updated in the event loop. I also implement set_ime_surrounding_text for Wayland and call it whenever it changed, especially when text input enabled (and commit it to compositor). But I still haven't receive any DeleteSurroundingText event triggered by compositor. Do you have any idea about what I'm missing?

Thank you .

kchibisov commented 6 months ago

But I still haven't receive any DeleteSurroundingText event triggered by compositor. Do you have any idea about what I'm missing?

it's IME dependent, you'll get it only when IME thinks that you should get it. Also, ensure to commit when setting surrounding text.

phnaharris commented 6 months ago

I understand. But if I reproduce the same situation in the same environment, I should expect the same result right? I've created a text box with gtk and egui (with winit), input the same content and when I go back to change some character, gtk worked fine with surrounding text while winit receive a new Preedit event.

Have you ever got a DeleteSurroundingText event in winit before? I just wanna know if the bug come from the flow that I'm implemented was wrong or something else?

My environment:

kchibisov commented 6 months ago

The event is not handled by winit unless you've added it. Use WAYLAND_DEBUG=1 to compare patters and use that knowledge to troubleshoot, since usually it's all pretty deterministic.

phnaharris commented 6 months ago

Okay, I got the idea. Thank you for your help.