Support Microsoft modern Japanese IME

[nishimotz]

Is your feature request related to a problem? Please describe.

Windows 10 Insider build already comes with the modern Japanese IME, which uses the same architecture as the Cloud Clipboard and the Emoji Panel.

NVDA already have AppModule for them, so modern Japanese IME does not work.

Describe the solution you'd like

I have worked around NVDA snapshot alpha 18399, and the Windows Insider build 18956.

Following app module gives the same experiences as with Japanese IME of Windows 10 version 1903. It breaks current support for Cloud Clipboard and Emoji Panel, though.

# windowsinternal_composableshell_experiences_textinput_inputapp.py
import appModuleHandler
import api
import speech
import braille

class AppModule(appModuleHandler.AppModule):

    def event_UIA_elementSelected(self, obj, nextHandler):
        s3 = obj.parent.parent.parent.UIAElement.cachedClassName + ":" + obj.parent.parent.parent.UIAElement.cachedAutomationID
        s2 = obj.parent.parent.UIAElement.cachedClassName + ":" + obj.parent.parent.UIAElement.cachedAutomationID
        s1 = obj.parent.UIAElement.cachedClassName + ":" + obj.parent.UIAElement.cachedAutomationID
        s  = obj.UIAElement.cachedClassName + ":" + obj.UIAElement.cachedAutomationID
        if (
            s3 == "Windows.UI.Core.CoreWindow:" and
            s2 == "pane:IME_Candidate_Window" and
            s1 == "ListView:IME_Candidate_Window" and
            s == "ListViewItem:"
        ):
            speech.cancelSpeech()
            api.setNavigatorObject(obj)
            speech.speakSpelling(obj.name, useCharacterDescriptions=True)
            braille.handler.message(braille.getBrailleTextForProperties(name=obj.name, role=obj.role, positionInfo=obj.positionInfo))
        nextHandler()

Describe alternatives you've considered

I am studying windowsinternal_composableshell_experiences_textinput_inputapp.py and understand that it is difficult to know which event comes from which application or input system. Because modern Japanese IME is still in preview, we should give feedback to Microsoft regarding this, such as to use different class names for them.

Additional context

The new IME once appeared in Windows Insider build 18277 last year, but it is not released with version 1903.

I am testing with the Fast Ring of Windows 10 Insider. However, it is reported by another user that the modern Japanse IME comes with Slow Ring as well, which is targeting version 19H2.

I have filed previous issues regarding Japanese IME such as #2730 and #2776. They should be nice to have with new IME as well. However, this issue is limited to give the support of the same level as current IME by NVDA.

[josephsl]

Hi,

In the past, I used build range to detect which part of modern keyboard is in use, but since the 19H1 PR, I have switched to looking at automation ID and build range.

If the CJK IME header uses a different automation ID and/or class name from emoji panel and clipboard history, then the solution becomes a bit simpler by checking for IME part and acting accordingly. If not, we may need to resort to build range.

One way to figure this out is restarting NVDA with debug logging enabled. If Windows 10 App Essentials add-on is installed, that add-on will print essential information about an object whenever UIA event comes along, namely class, name, event, automation ID, app, and class name. At the moment it does not print the automation ID for the actual modern keyboard header, but I'm willing to add that change with the next WinTenApps development snapshot.

Thanks.

[nishimotz]

As shown in my experimental code, most class names and automation IDs are the same among Japanese IME, Emoji Panel, and Cloud Clipboard, within the same Windows build. I am afraid Windows build number does not help.

[josephsl]

Hi,

Many updates since then...

Compounding the problem is that the CJK IME in 20H1 is hosted inside Text Input Host, the same executable as modern keyboard. So far it works with Windows 10 App Essentials add-on installed, but we do need to prepare for this going to NVDA Core in say, 2020.1 (preferably 2019.3 but that can be changed).

Technical: when CJK IME candidates window opens, a different UIA automation ID comes up: IME_Candidate_Window. This means we can do something about it, including announcing the first selected candidate. This simplifies the solution a lot. Because of this, this will be part of PR #9205 with a priority boost.

I'll cook up a PR build for you to test in a few hours.

Thanks.

[josephsl]

Hi,

Another observation: UIA tree structure is similar to hardware keyboard suggestions, which makes the solution even simpler.

Thanks.

[josephsl]

Hi,

Can you try this PR build and see if it resolves this for you?

Thanks.

[josephsl]

Hi,

Upon further testing, it appears it might be possible to tweak my solution a bit - the PR build I linked supports Hanja, not actual Japanese IME. A new PR build will be sent shortly.

Thanks.

[josephsl]

Hi,

Although I'm going to include Japanese IME support in Windows 10 App Essentials, I'll add your solution (spelling the candidate item) after the modern keyboard enhancements are merged into Core first (i.e. I'm reverting IME support from the parent PR for now). The best way to incorporate your solution is the moment the candidate item is set as navigator object, and can be done in as few as two lines.

Thanks.

[josephsl]

Hi,

Another observation: modern IME isn't part of 19H2 - I tested it with Korean and Japanese input in build 18363.

Thanks.

[josephsl]

Blocked by #10359. Modern CJK IME support will be a separate PR, not to be done as part of #9205, which I'm thinking of withdrawing due to scope creep.

Thanks.

[larry801]

Tried this patch in new Chinese IME from build 18990.1

AutomationId of Candiate panel turn out to be "IME_Prediction_Window"

After changing that Id ,NVDA reads the first candidate, but if I try to switch to other candidates.It does not work.

After using accevent.exe I find out the following things.

There are no UIA element selected for new Modern Chinese IME Candidate.

There are only out-of-context winEvents which are not tracked by NVDA.

Previous version of ime emits in context selection event and UIA element selected event.

[josephsl]

Hi,

10359 is now part of NVDA.

Adding to the urgency: Windows 10 build 19536 (fast ring) re-introduces modern Korean IME. However, work associated with it is suspended until after NVDA 2019.3 is released (I'll investigate it further as part of Windows 10 App Essentials add-on, however).

Thanks.

[josephsl]

Potentially fixed with #11562 PR. Thanks.

[nishimotz]

Testing #11562 snapshot with Japanese IME.

setup:

• nvda_snapshot_pr11562-20880,f8f4a842.exe from https://ci.appveyor.com/project/NVAccess/nvda/builds/35123074/artifacts
• Windows 10 Pro version 20H2 build 19042.508 (x86 Japanese)
• Microsoft (modern) IME with default settings
• NVDA Input Composition settings: Automatically report all available candidates is turned off
• built-in Focus Highlight is turned on
• no add-ons installed or enabled
• other NVDA options are not modified from the default

steps and issue 1

• press [Fullshape/Halfshape] key to start the composition (Latin-to-syllable Transliteration a.k.a. Romaji)
• type "k"
• red rectangle (navigator object) is moved to the "predictive input" popup. All prediction items are sent to the synthesizer. Only the last prediction item can be heard. As the result, the announce regarding Latin-to-syllable Transliteration, which is expected, can not be heard.

steps (continued) and issue 2

• type "a" "w" "a" (kawa = river in Japanese)
• press space twice, popup for the candidate list is shown
• press space for moving to the next candidate item. item is announced, however, descriptions of each characters are not used.

notices

• sometims character descriptions are announced as expected. it depends on the text which is under transliteration.
• in case of "yama" (mountain in Japanese) : candidate items are announced as expected
• in case of "kanji" (Chinese character in Japanese) : at the begining, it works as expected. however, if 9th candidate item or later is selected, the popup window is expanded to 4 columns. after that, it does not work as expected.

[josephsl]

Hi, I strongly suspect event handling issue, along with a possibility that Japanese IME (Romaji, for example) candidate items are not properly recognized as an IME item. To at least confirm this, can you try using the latest development snapshot of Windows 10 App Essentials (which does include the mentioned PR in a slightly modified form), restart with debug logging enabled, and send the debug log to Mick and I (or attach it here and on #11562)? That way we can figure out what can be done in a later iteration of the PR. Thanks.

[Adriani90]

@nishimotz, @josephsl now that #11562 is merged, is there improvement regarding this issue visible?

[josephsl]

Hi,

Complicating matters is that Windows 11 changed parts of the CJK IME interface, so looks like an updated PR might be needed next year.

Thanks.

[josephsl]

Hi,

2022 update: looks like Windows 11 Version 22H2 keeps the basic IME mechanism and UIA implementatino as in Windows 11 Version 21H2, and I'm working on transforming changes introduced in Windows App Essentials add-on into a reviewable pull request. Note that due to focus events, the PR wil require some explanation, therefore I'l create a new issue just to handle Windows 11 IME.

Thanks.

[josephsl]

Hi,

March 2024 update: anyone experiencing problems in 2024? If yes, I advise waiting until #16313 is merged before revisiting Japanese IME problem (across Windows 10 and 11).

Thanks.

[Adriani90]

cc: @nishimotz could you test this with NVDA last alpha, or after the PR refereced by Joseph above is merged?