Can we make mutation to DOM cancelable during IME?

[chong-z]

Background Originally I proposed https://github.com/w3c/input-events/issues/34 because I was under the impression that the Editing TF has already made the conclusion that we cannot do anything to composition text. However @yosinch reminded me that we haven't yet got a conclusion.

Two Proposals After digging into mailing list I found @ojanvafai 's email[1] about two proposals:

1. Spec UA's behavior during IME, assuming they aren't always preventDefaultable.
   • This is what we are currently doing
2. @rniwa 's proposal[2]: Make the mutation to DOM cancelable (But IME state remains untouched)
   • Could also be applied to cE=true
   • UA may decide to cancel IME session if JS failed to update DOM with expected composition text

I'm not sure what's the outcome from TPAC 2015's discussion and I cannot find minutes / resolutions (Edited: Here is the meeting notes[3] for the breakout session, but I can't find discussion about @rniwa 's proposal).

So what's the reason we didn't go with proposal 2?

[1] https://lists.w3.org/Archives/Public/public-editing-tf/2015Oct/0117.html [2] https://lists.w3.org/Archives/Public/public-editing-tf/2015Oct/0085.html [3] https://lists.w3.org/Archives/Public/public-editing-tf/2016Jan/0002.html

[johanneswilm]

Simplicity. Giving JS the opportunity to do the DOM update is only helpful for us if we can diverge from what the browser would have done. But if the IME cancels on us because we didn't do exactly what the browser would have done anyway, then there is not much of a point with giving JS control over the DOM update. Also, I don't think it was clear to us what can and cannot be done during IME. At one stage we heard that a new Microsoft IME will crash if you move the selection during composition start. At another point we heard that it is probably ok as long as it keeps the same offset. Having more control over the DOM updates during IME would be great, but it would need to mean that we could really do the DOM updates we want to do without having the IME crash on us. The current solution gives us a good way of dealing with IME once it's done. The hat's a big step forward.

[johanneswilm]

Reading through the debate from back then. I wonder if we can:

1. Combine Ryosuke's proposal for cancelable IME with Chong's proposal for an inputtype to remove the original composition string in case of recomposition.
2. Ensure that the composition is not canceled, no matter how JS changes the DOM (a composition box outside the editing host should be allowed).
3. Be sure that only composition text is underlined. The JS editor may decide to change other parts of the DOM inside the editinghost (fx in an collaborative editor) while the composition takes place, and this should not be underlined.
4. Have this work on all UA/IME combinations.

Point 4 seems to be the most problematic or the one we are the least sure of. Point 3 may also be difficult and it may be easier to give JS the ability to style any range.

No matter whether or not we do this, it may be best to wait for the next version, as this introduces a number of new complications and given the way IME iscurrently handled, it shouldn't be hard to write JS code that works with both versions and gradually proves tge user experience as the new version becomes available.

[chong-z]

Thanks @johanneswilm for sending me the note! I've updated issue description (but still can't find discussion about proposal 2).

@rniwa what's your current opinion on this issue, do you still have the proposal in mind or do you want to leave it for the next version?

But if the IME cancels on us because we didn't do exactly what the browser would have done anyway, then there is not much of a point with giving JS control over the DOM update.

I think for V1 the best thing JS can do is to update composition text with custom style. e.g. JS can insert "bo<b>l</b>d" if they were asked to insert "bold" (but the characters should be exactly the same). Will that be a good motivation?

At one stage we heard that a new Microsoft IME will crash if you move the selection during composition start.

IIRC @masayuki-nakano has some comments about TSF on Windows, do you think proposal 2 + adding custom style is doable on FireFox?

At another point we heard that it is probably ok as long as it keeps the same offset.

Including the mentioned offset issue, if we want to go with proposal 2 we will at least need:

1. A way to specify caret offset in 'beforeinput'
2. A plan to coordinate markers. e.g.
   • UA will add markers starting from caret position for the expected composition text, or
   • JS needs an API to add markers
3. Specify what JS can do during these event handlers
   • Which APIs can be used to update composition text
   • Can they move focus, modify other part of the DOM, or do rendering using their internal data model?

@yosinch I believe you are in favor of proposal 2, do you think we can figure out the above issues in V1?

Combine Ryosuke's proposal for cancelable IME with Chong's proposal for an inputtype to remove the original composition string in case of recomposition.

The combined proposal should be having only one inputType 'insertCompositionText' for IME, which is cancelable and covers all 'compositionstart/update/end' operations.

Ensure that the composition is not canceled, no matter how JS changes the DOM (a composition box outside the editing host should be allowed).

I doubt if we can do it.

[johanneswilm]

I think for V1 the best thing JS can do is to update composition text with custom style. e.g. JS can insert "bo<b>l</b>d" if they were asked to insert "bold" (but the characters should be exactly the same). Will that be a good motivation?

This will likely cover most cases, but probably not all. What about spaces, for example. Do they count as letters? What about ligatures for "fl" and "ll"? What if an editor uses images or SVGs for certain letters Can we insert those without breaking the IME?

For example, if I want to insert an empty and non-editable span for styling purposes on the middle of the text, I normally need to put a space inside of it so that the caret moves around it correct in---among other browsers---Chrome (<div contenteditable="true">Hello<span contenteditable="false">&nbsp;</span> there!</div>) One can use a zero-width space instead, but I guess that will also count as a character, so if one does this during IME composition, one will need to first add an empty span, then wait for the composition to finish, and then add the space... It's not impossible to do, but it just complicates things and doesn't make IME something that is easily handled by JS.

Can they move focus, modify other part of the DOM, or do rendering using their internal data model?

No matter what solution we choose, knowing what can and cannot be done to the DOM during composition would be valuable information for JS editor developers.

Combine Ryosuke's proposal for cancelable IME with Chong's proposal for an inputtype to remove the original composition string in case of recomposition.

The combined proposal should be having only one inputType 'insertCompositionText' for IME, which is cancelable and covers all 'compositionstart/update/end' operations.

How do we then differentiate between the different parts of the composition? On the part of the JS editor, we may want to record the dom structure at the time of what is currently "deleteByComposition" in order to try to apply those styles after the composition has ended. If we only have one, that can be canceled for some things and not for others, we will still need to record things before the composition starts so that we can then try to reapply the style after the composition has ended.

---

I am a little skeptical trying to figure this out now for version 1. We have experienced several times that we thought something could be easily fixed, and in the end it turns out it is much more complicated and we never get to something that works well enough because everything is more complex than what it seemed to be initially. I've only been in thee W3C process for about 2 years, and followed the issue for 4 years, but I think others have worked on this for 8-11 years and maybe even longer. I don't think we have come so close to a working solution as we have now, and there are many other issues we would also like to fix, so we will have to keep changing contenteditable anyway. It would be really sad if we miss it because we thought we could fix one more thing that turns out to be a lot more complicated.

That being said, the current situation may be different than the previous ones. Could we maybe say something like "If this just works with no further complications at all, we can add it to the current version. But if we run into the slightest issue, we keep what we have right now and put everything else off to the next version." Would that be OK?

[chong-z]

Could we maybe say something like "If this just works with no further complications at all, we can add it to the current version. But if we run into the slightest issue, we keep what we have right now and put everything else off to the next version." Would that be OK?

Yes that's definitely the agreement I(personally)'d like to see, but I want to make sure everyone is on the same page since I've already got objections from @alexelias and @yosinch. Anyway will wait for other inputs.

[johanneswilm]

I've already got objections from @alexelias and @yosinch.

What are these objections about? The only object by @alexelias that I have seen here was retracted: https://github.com/w3c/input-events/issues/31#issuecomment-257799800

[chong-z]

What are these objections about? The only object by @alexelias that I have seen here was retracted: #31 (comment)

@alexelias 's objection can be found here crbug.com/652397, and @yosinch is in favor of proposal 2. But anyway we need to wait for their current opinion.

[yosinch]

My position is support proposal 2; All DOM mutation is cancelable and "deleteByComposition" is redundant since "insertFromComposition" can delete/insert once.

From rendering engine perspective IME is data manager managing:

• Composition string; active and inactive text == characters with active/inactive attribute
• Insertion point, aka caret, cursor

IME notifies its own data change to rendering engine by composition{start,update,end} event. And, IME queries view port coordinate of insertion point for displaying IME UI, e.g. candidates window, toolbox.

So, responsibility of rendering engine is provide displaying IME state to user. JS can cancel "insertFromComposition" and mutation DOM whatever JS wants.

Once, we expose composition text attributes, JS can display composition by itself, e.g. changing background color instead of "default" black thick/thin underline.

BTW, Blink has an issue when composition is occurred in a element with text-transform, which changes number of characters, e.g. "ß" (Eszett) => "SS". But this issue doesn't make IME in trouble, since IME manages its own state. This is only UI issue.

[chong-z]

So, responsibility of rendering engine is provide displaying IME state to user. JS can cancel "insertFromComposition" and mutation DOM whatever JS wants.

@yosinch I can see it should be eventually doable, but do you think there will be any technique difficulties for the implementation? e.g.

1. Some IME may query text data and terminate if it doesn't match it's internal state, how do we handle that?
2. How do we specify the caret offset inside composition text?
3. What's your opinion on handling underlines? I believe @rniwa is in opposite position of giving JS the control[1]
   • e.g. How do they apply underlines in respect of word boundaries and providing a consistent UX?
4. Do we want to spec what JS can / can't do during these event handlers?
   • e.g. Can they put images inside composition text?

The original plan was to finish implementation by EOY, so how long would you expect to take for specing / implementing proposal 2?

Also suppose everyone is in favor of propoal 1, do you want to block on that or are you OK with going with proposal 1 first and figuring out proposal 2 later?

[1] https://lists.w3.org/Archives/Public/public-editing-tf/2015Oct/0112.html