IME testing

[jgraham]

Behaviour of IME events has been identified as a specific pain point for library authors and implementors. Currently we don't have a good way of writing tests against IME composition events. As part of this investigation we should:

• Propose an extension to the WebDriver specification that will allow writing IME tests
• Add IME support to testdriver so that it can be used with web-platform-tests
• Write some tests using the new API

The aim is that the identified tests would be a candidate for an Interop 2023 focus area that browsers would be able to commit to.

[jgraham]

Please let me know if the summary as written isn't right, doesn't cover something important, or covers something it shouldn't.

[jgraham]

I've filed https://github.com/w3c/webdriver/issues/1668 as an issue on the WebDriver spec here. I plan to write that up as a proper WebDriver proposal and also add integration into web-platform-tests / WebDriver.

[jgraham]

I suggest that in the first instance we set the goal here as having a proposal for IME input in the WebDriver standard, and meaningful progress on getting it approved, so that we can write a web-platform-tests RFC for including it in testdriver. I'm confident that if we get to that stage we'll also have a gecko implementation, but I don't think we should consider that, or writing new tests, to be part of the metric. However I think they are good stretch goals, and will be needed if we want to make a specific Interop 2023 proposal.

[jgraham]

I've made an initial high level proposal at https://github.com/w3c/webdriver/issues/1683 No spec text, but an API proposal for how IME can fit into WebDriver actions. I would appreciate review on whether this makes sense (I'm not an IME expert!), and plan to discuss it with the Browser Testing & Tools WG at TPAC.

CC: @megangardner @whsieh @BoCupp-Microsoft @johanneswilm @dlibby- @masayuki-nakano

[jav099]

Hello!

I worked in this space as an intern for Microsoft Edge last year. We did some work upstream to enable automated testing for simulated IME composition events. This is the merged Chromium CL to enable a way for other libraries such as Playwright and Selenium to simulate IME input. We also did some work in Playwright for demo purposes, and looking at the Playwright PR that I worked on here it seems like they continued the work with this PR. It seems however to have been closed due to Playwright “not being ready for ime API at this point”. I do not remember all the details on our proposed API shape and so I am unsure of whether it lines up completely with yours, but since we had SetComposition (which included events for compositionEnd, compositionStart, compositionUpdate) then it probably is a good starting point to build on top of.

Nevertheless, with our Playwright PR alongside with the upstream changes we made last year we were able to get some basic IME (simulated) testing demoed, with the correct events getting fired within the browser. I am CCing @alexkeng since we worked on this together.

I can’t find my slides and demo video of the testing using our Playwright PR since they were made with my other Microsoft account that I have no access to anymore, but I’m thinking maybe Alex does since we had a shared OneNote with all of this?

Let me know if there are any questions on any of this and if I can help further 😊

Javier

[jgraham]

There is still an open PR here, but it hasn't received any substantive review. I suggest that PR being open is worth about 1/3 of the score in this area; getting it reviewed and the comments addressed so it can actually land would be the remaining 2/3. It's pretty clear that implementation is not going to happen this year (and we've missed the chance for that to be included in Interop 2023, since the spec wasn't in a good enough shape).

[jgraham]

Updating the score here to 30% per the meeting.

Next step is review of the WebDriver IME PR. If we feel like we can't assess that properly without a demo implementation then we that's also useful feedback, and should be noted in the review, so we can adjust the plan for completing this area.

[jgraham]

No further progress so keeping the 30% score.