IME tracking issue

[raphlinus]

This is the tracking issue for Input Method Editor (IME) support in Druid. As of the time of writing, it's not an immediate priority, but increasingly other things depend on it, and we've collected a fair amount of information on the topic. Also note, #267 has some info on this topic, but we'll go into more depth here. For more color on IME, the blog Text Editing Hates You Too makes good reading.

Cross-platform abstraction

The main work is creating a cross-platform abstraction for IME, which would be implemented in druid-shell. The main function of this abstraction includes:

• Allowing the IME to perform editing operations such as inserting text, changing the composition region, etc.

• Giving the IME enough context of the text being edited to be useful.

• Negotiating geometry so the IME can be displayed at the correct place on the screen. In general, the IME is displayed in its own window, not directly managed by the app. (See #1301, which is a manifestation of this requirement)

• Various lifecycle events and also dealing with language changes.

TextBox integration

The cross-platform abstraction lives in druid-shell. Layered on top of that is support for IME-driven editing in the TextBox widget. The work can in theory be split into two separate subprojects, but it might be difficult to validate the shell work without some kind of working text editing widget.

There are also decisions to be made whether to plumb the raw druid-shell IME interface up so (potentially user-written) widgets can use them, or whether to do some wrapping at the druid layer to take care of boilerplate and provide a somewhat higher level interface. For keyboard events, we do more of the former, but there is plenty of precedent for the latter as well. A reasonable rule of thumb is that anything platform specific (and thus unsafe) belongs in shell, but if there is logic that is common across most platforms, that belongs more in druid.

Existing work

There are two Rust GUI toolkits that have at least some IME support. The most complete appears to be tcw3, which is part of the Stella2 repo. The cross-platform abstraction is in iface.rs, but IME is not split out. The major relevant traits are Wm::new_text_input_ctx, Wm::text_input_*, TextInputCtxListener, and TextInputCtxEdit. Look under tcw3/pal/src/*/textinput for the implementations of those traits. The tcw3 subdirectory is MIT-licensed. If we feel we want to directly use any of that code, it should be possible to iron out any licensing related kinks.

There is also some IME support for macOS in Makepad. One relevant type is TextInputEvent, plus lots of implementation in cx_cocoa.rs and cx_win32.rs.

One source I don't expect to be especially useful is winit. They do have some implementation of IME interfaces (particularly on macOS), but architecturally it's based on synthesizing key events based on the inserted text, which is wrong.

There are other IME abstractions worth studying. Browsers are always a rich source. In particular, the Mozilla IME handling guide provides excellent detail on what proper app behavior should look like.

There is a proposed Web Input Method Editor API, but it looks like that stalled out and is not implemented by any current browser. There is some reporting of IME editing from text input areas to applications, including the compositionupdate event and IME-related events such as insertCompositionText and insertReplacementText as part of Input Events.

Platform-specific considerations

Windows

The main IME interface on Windows is Text Services Framework. That in turn defines a large number of interfaces.

One concern on Windows is that Text Services Framework requires instantiation of COM interfaces such as ITfContextOwnerCompositionSink. Ultimately, we want to use com-rs for this, but it has not yet stabilized. Both tcw3 and Makepad hand-roll the COM implementation, directly writing out the vtables. That's probably good enough for the interim.

macOS

IME on macOS is important for a number of reasons. For one, it is the only recommended way to get dead-key handling. By contrast, Windows will apply dead-key logic at the raw keyboard level. IME support is also necessary for correct operation of the the dictation and emoji menu items, which should appear under "Edit" (see #1307).

The main interfaces are NSTextInputClient and NSTextInputContext. As is a typical split, the first is implemented by the app to recieve input from the IME, and the second is called by the app to supply the IME with context.

There is a small amount of tweaking still needed at the raw keyboard level. The dead key should produce KbKey::Dead, which requires some low-level processing. Browsers use UCKeyTranslate for this, which is not recommended, but I don't believe there is a real way to do this using only officially recommended APIs. Do see #1040 for more detail on this particular issue.

Also note, tcw3 uses a substantial amount of Objective-C code to do the platform binding. We've tended do this binding work on the Rust side, using msg_send! and add_method of our own extern "C" methods to classes that we need to instantiate. This ties to much deeper questions. In my opinion, the Objective-C interop story in Rust suffers from a number of problems and doesn't currently have a strong maintainer, so moving more logic to Objective-C as tcw3 has done is appealing.

I haven't investigated iOS yet, but would expect it to be broadly similar to macOS.

Linux

Following a common pattern, the world is somewhat different if we're depending on Gtk as opposed to doing direct X11 or Wayland integration. Gtk has a perfectly fine IME interface, based on GtkIMMulticontext and GtkIMContext. For the X11 (or Wayland) case, we'd have to make some decisions where to bind to platform capabilities and which variants of those to support. This is no doubt a major reason Makepad does not support IME on Linux.

That said, winit does have some IME support on X11, and it might be worth taking a look.

Android

Supporting IME is one of the major blockers to using Druid on Android. That's not in scope for this issue, but we should be aware of it, and make design decisions that won't paint us into a corner later.

The main Android interfaces are InputMethodManager and InputConnection.

I should also point out that Android's IME is a bletcherous mix of async and synchronous methods. It is also very confused about whether it's low level (some of the edits are specified in terms of UTF-16 code units) or slightly higher level (deleteSurroundingTextInCodePoints is at least code points rather than code units, so can't split an astral plane character).

Rust ecosystem

All non-toy GUI toolkits will need to support IME at some point. Since there may be other Rust toolkits that aspire to this status, it makes sense to at least consider possibilities for collaboration, or making a crate to support IME that is not quite as tightly bound to druid-shell.

One possible inspiration is keyboard-types, which is a "vocabulary" crate setting out a well-designed type for raw keyboard events, separated out from any implementation. It's possible to imagine something similar for IME, though it's a lot more complex. It's also possible that correct wiring up of lifecycle will depend on specifics of the event loop, and there might be assumptions about threading (in Druid we can safely assume the UI eventloop is single threaded).

That said, keyboard-types has not had much adoption. It is (was?) used in Servo, but layered on top of a crappy winit-based implementation, so didn't work well there. It is being considered in winit but that discussion is so far inconclusive - they seem to be trying to reinvent what we've already done in Druid-shell, but I'm not sure.

So probably a good thing to do is circulate this tracking issue to other people in the Rust GUI space and see if they have any interest in coordinating work. If not, we can confidently proceed on our own.

[zjiekai]

A blog post about X11 based IME https://tedyin.com/posts/a-brief-intro-to-linux-input-method-framework/

[maroider]

Just in case you missed it: The main discussion on IME support in Winit seems to be in rust-windowing/winit#1497 now (as a continuation of rust-windowing/winit#1293).

[lord]

still have a long way to go, but have started prototyping input method API ideas for druid-shell in a branch! half-works for pinyin input on macos, with occasional crashes because i have yet to convert code unit lengths from utf-8 to utf-16 ;) try it out in the branch with cd druid-shell; cargo run --example ime

right now I'm just implementing NSTextInputClient directly on the main app view, which seems wrong if there are multiple text fields?

[raphlinus]

I'm not sure about the scope of NSTextInputClient - it seems to me it's not necessary to persist state across text fields when switching focus, so maybe it's ok? In any case, great to see this progress.

[Riey]

I recently implemented the XIM protocol So maybe it could be helpful for x11 platform.

[quark-zju]

For Windows, it's possible to use Win32 APIs like ImmSetCompositionWindow to add integration. COM interface does not seem necessary.

[H-M-H]

I recently implemented a minimal wrapper https://github.com/H-M-H/xcb-imdkit-rs/ for https://github.com/fcitx/xcb-imdkit which implements the XIM protocol as well. It is being used for https://github.com/wez/wezterm now. As far as I can tell druid switched from xcb to x11rb recently, so if there is interest I can add support for that as well as my crate only works with xcb at the moment.

[Eschryn]

For Windows, it's possible to use Win32 APIs like ImmSetCompositionWindow to add integration. COM interface does not seem necessary.

From what I understand TSF replaces and deprecates IMM32

And IMM is more focused on IMEs (from what I gather) while TSF is much more abstract and is also for Handwriting, Speech etc. Input

[Eschryn]

Currently working on the Windows IME support and im making progress.

[ArcticLampyrid]

Currently working on the Windows IME support and im making progress.

Hi, is there any progress? For now the position of the composition window is incorrect on Win32, which means it's almost unuseable for CJK users.

[Eschryn]

Currently working on the Windows IME support and im making progress.

Hi, is there any progress? For now the position of the composition window is incorrect on Win32, which means it's almost unuseable for CJK users.

Yeah - but i havent gotten around to fixing the bugs yet

https://github.com/Eschryn/druid/tree/feature/tsf-implementation