EditContext: Do IME see the plaintext in `EditContext`?

[masayuki-nakano]

I don't find this information. As you konw, modern IME on desktop retrieve text and selection for improving suggestions and candidate list. So, returning same text to IME is important for compatibility between browsers.

And also it's important information for IME on desktop where is the caret or the selected clause of composition string to show their own window. However, EditContext does not have this information.

[masayuki-nakano]

Oh, I find updateBounds(). However, it takes only the editor rect and a range rect of selection. But I think that it's not enough for Japanese IME. Composition string of Japanese IME may be long so that sometimes wrapped and stay in multiple lines. And it may have multiple clauses so that IME may request a rect for specific clause (or character).

[alexkeng]

yes IME do see the plaintext in EditContext, so the suggestion in software keyboard or candidate window in CJK will still work with EditContext.

As for character bounds, it does sound like a missing API, we'll evaluate and add it to the spec, thanks!

[masayuki-nakano]

One difficult thing of storing character bounds is, which rects of characters should be stored. As you know, the main process of a browser interacts with native IME, but layout information (in this case, EditContext) is in a remote process). So for making their communication asynchronous, layout information is cached in the main process before native IME retrieves character rects.

E.g., Gecko caches the following character rects in the main process:

• All characters' rects in composition string if in a composition (required for showing candidate list window, some IME may show it at start of composition string, some other IME may show it at start of selected clause in composition string, etc)
• The first character rect in the editor (if there is no text, stores the caret rect for inserting first character)
• Before/after character rects of anchor/focus point of Selection (This is required for allowing IME to show tooltip like UI around selection)
• All characters' rects at last commit string only immediately after composition ends (this is required for Kakutei-undo feature of Japanese IMEs)

Once those requirements from browser is defined in the spec, web apps may be able to compute the character rects with simpler code (or they can put the layout information actively). And if the text position is changed by scroll in the view, those information need to be updated. So this issue may cause performance issue if web apps computes the rects too much or too strictly.

[alexkeng]

So for making their communication asynchronous, layout information is cached in the main process before native IME retrieves character rects.

Blink is also caching layout infomation in the main process, but with little difference:

1) All characters' rects in composition string if in a composition (required for showing candidate list window, some IME may show it at start of composition string, some other IME may show it at start of selected clause in composition string, etc)

same

2) The first character rect in the editor (if there is no text, stores the caret rect for inserting first character)

Blink only caches rect of current selection, (or rect of the caret if no selection). What is the use of the first char rect for IME if the char is not in active composition?

3) Before/after character rects of anchor/focus point of Selection (This is required for allowing IME to show tooltip like UI around selection)

do you mean if the string is "A|BCD|EF" and selection is (1,4), Gecko caches rect for 'A''B' and 'D''E'? in this case Blink caches rect for 'BCD', is that not enough for IME?

4) All characters' rects at last commit string only immediately after composition ends (this is required for Kakutei-undo feature of Japanese IMEs)

could you elaborate more? we already cache all chars in composition in 1) right? do you mean after comosition ends, we don't reset? and what is Kakutei-undo feature? how do we apply it?

[masayuki-nakano]

1. The first character rect in the editor (if there is no text, stores the caret rect for inserting first character)

Blink only caches rect of current selection, (or rect of the caret if no selection). What is the use of the first char rect for IME if the char is not in active composition?

According to the log, it was required for supporting Japanese IME on macOS Yosemite. https://bugzilla.mozilla.org/show_bug.cgi?id=1171814

1. Before/after character rects of anchor/focus point of Selection (This is required for allowing IME to show tooltip like UI around selection)

do you mean if the string is "A|BCD|EF" and selection is (1,4), Gecko caches rect for 'A''B' and 'D''E'? in this case Blink caches rect for 'BCD', is that not enough for IME?

It was supporting for query rect of previous character when caret is put at end of line. I don't remember the detail though. https://bugzilla.mozilla.org/show_bug.cgi?id=1257446

1. All characters' rects at last commit string only immediately after composition ends (this is required for Kakutei-undo feature of Japanese IMEs)

could you elaborate more? we already cache all chars in composition in 1) right? do you mean after comosition ends, we don't reset? and what is Kakutei-undo feature? how do we apply it?

Kakutei-Undo is that restore last composition string which is committed immediately before (i.e., if you move selection or typing other things, it may become not available). At this time, IME queries character rects of the last commit string before or immediately after starting new composition. I.e., this is required when there is no active composition, but Kakutei-Undo is available.

[alexkeng]

@masayuki-nakano we have added a new event CharacterBoudnsUpdateEvent and new API updateCharacterBounds, where CharacterBoundsUpdateEvent describes what character bounds are needed by the Text Input Service. After receiving CharacterBoundsUpdateEvent, the author is responsible to compute the requested character bounds and call updateCharacterBounds to update the character bounds cached in the EditContext. (please see the example 3 in the spec for more details)

Do you think this API is sufficient to cover your 4 scenarios? or do you think we'll need an event/API that support an array of ranges? note we have updateSelectionBound API that can let the author set the caret bound, so it should cover your 2nd scenario when there is no text.

also do you know how to trigger Kakutei-Undo on Windows?

[dandclark]

I think the questions in this issue were resolved in Alex's response here; see CharacterBoundsUpdateEvent and https://w3c.github.io/edit-context/#editcontext-events in the spec. Feel free to reopen if there are still unresolved issues with this.