Open nishimotz opened 5 years ago
Hi,
In the past, I used build range to detect which part of modern keyboard is in use, but since the 19H1 PR, I have switched to looking at automation ID and build range.
If the CJK IME header uses a different automation ID and/or class name from emoji panel and clipboard history, then the solution becomes a bit simpler by checking for IME part and acting accordingly. If not, we may need to resort to build range.
One way to figure this out is restarting NVDA with debug logging enabled. If Windows 10 App Essentials add-on is installed, that add-on will print essential information about an object whenever UIA event comes along, namely class, name, event, automation ID, app, and class name. At the moment it does not print the automation ID for the actual modern keyboard header, but I'm willing to add that change with the next WinTenApps development snapshot.
Thanks.
As shown in my experimental code, most class names and automation IDs are the same among Japanese IME, Emoji Panel, and Cloud Clipboard, within the same Windows build. I am afraid Windows build number does not help.
Hi,
Many updates since then...
Compounding the problem is that the CJK IME in 20H1 is hosted inside Text Input Host, the same executable as modern keyboard. So far it works with Windows 10 App Essentials add-on installed, but we do need to prepare for this going to NVDA Core in say, 2020.1 (preferably 2019.3 but that can be changed).
Technical: when CJK IME candidates window opens, a different UIA automation ID comes up: IME_Candidate_Window. This means we can do something about it, including announcing the first selected candidate. This simplifies the solution a lot. Because of this, this will be part of PR #9205 with a priority boost.
I'll cook up a PR build for you to test in a few hours.
Thanks.
Hi,
Another observation: UIA tree structure is similar to hardware keyboard suggestions, which makes the solution even simpler.
Thanks.
Hi,
Upon further testing, it appears it might be possible to tweak my solution a bit - the PR build I linked supports Hanja, not actual Japanese IME. A new PR build will be sent shortly.
Thanks.
Hi,
Although I'm going to include Japanese IME support in Windows 10 App Essentials, I'll add your solution (spelling the candidate item) after the modern keyboard enhancements are merged into Core first (i.e. I'm reverting IME support from the parent PR for now). The best way to incorporate your solution is the moment the candidate item is set as navigator object, and can be done in as few as two lines.
Thanks.
Hi,
Another observation: modern IME isn't part of 19H2 - I tested it with Korean and Japanese input in build 18363.
Thanks.
Blocked by #10359. Modern CJK IME support will be a separate PR, not to be done as part of #9205, which I'm thinking of withdrawing due to scope creep.
Thanks.
Tried this patch in new Chinese IME from build 18990.1
AutomationId of Candiate panel turn out to be "IME_Prediction_Window"
After changing that Id ,NVDA reads the first candidate, but if I try to switch to other candidates.It does not work.
After using accevent.exe I find out the following things.
There are no UIA element selected for new Modern Chinese IME Candidate.
There are only out-of-context winEvents which are not tracked by NVDA.
Previous version of ime emits in context selection event and UIA element selected event.
Hi,
Adding to the urgency: Windows 10 build 19536 (fast ring) re-introduces modern Korean IME. However, work associated with it is suspended until after NVDA 2019.3 is released (I'll investigate it further as part of Windows 10 App Essentials add-on, however).
Thanks.
Potentially fixed with #11562 PR. Thanks.
Testing #11562 snapshot with Japanese IME.
setup:
steps and issue 1
steps (continued) and issue 2
notices
Hi, I strongly suspect event handling issue, along with a possibility that Japanese IME (Romaji, for example) candidate items are not properly recognized as an IME item. To at least confirm this, can you try using the latest development snapshot of Windows 10 App Essentials (which does include the mentioned PR in a slightly modified form), restart with debug logging enabled, and send the debug log to Mick and I (or attach it here and on #11562)? That way we can figure out what can be done in a later iteration of the PR. Thanks.
@nishimotz, @josephsl now that #11562 is merged, is there improvement regarding this issue visible?
Hi,
Complicating matters is that Windows 11 changed parts of the CJK IME interface, so looks like an updated PR might be needed next year.
Thanks.
Hi,
2022 update: looks like Windows 11 Version 22H2 keeps the basic IME mechanism and UIA implementatino as in Windows 11 Version 21H2, and I'm working on transforming changes introduced in Windows App Essentials add-on into a reviewable pull request. Note that due to focus events, the PR wil require some explanation, therefore I'l create a new issue just to handle Windows 11 IME.
Thanks.
Hi,
March 2024 update: anyone experiencing problems in 2024? If yes, I advise waiting until #16313 is merged before revisiting Japanese IME problem (across Windows 10 and 11).
Thanks.
cc: @nishimotz could you test this with NVDA last alpha, or after the PR refereced by Joseph above is merged?
Is your feature request related to a problem? Please describe.
Windows 10 Insider build already comes with the modern Japanese IME, which uses the same architecture as the Cloud Clipboard and the Emoji Panel.
NVDA already have AppModule for them, so modern Japanese IME does not work.
Describe the solution you'd like
I have worked around NVDA snapshot alpha 18399, and the Windows Insider build 18956.
Following app module gives the same experiences as with Japanese IME of Windows 10 version 1903. It breaks current support for Cloud Clipboard and Emoji Panel, though.
Describe alternatives you've considered
I am studying windowsinternal_composableshell_experiences_textinput_inputapp.py and understand that it is difficult to know which event comes from which application or input system. Because modern Japanese IME is still in preview, we should give feedback to Microsoft regarding this, such as to use different class names for them.
Additional context
The new IME once appeared in Windows Insider build 18277 last year, but it is not released with version 1903.
I am testing with the Fast Ring of Windows 10 Insider. However, it is reported by another user that the modern Japanse IME comes with Slow Ring as well, which is targeting version 19H2.
I have filed previous issues regarding Japanese IME such as #2730 and #2776. They should be nice to have with new IME as well. However, this issue is limited to give the support of the same level as current IME by NVDA.