refactor(web): active element management in relation to OSK display control ✂️

[jahorton]

Addresses aspects of #5026.

Follows #5633.

Fixes #5026. (Finally!) This PR finally centralizes management of KMW's active element logic and fixes a few niche scenarios that went relatively undetected until now.

One of the worst aspects of "spaghetti code" in relation to the OSK is how entangled some of its display aspects have been with the web engine's active-element management. While this PR doesn't get everything cleaned up perfectly for active-element tracking, its biggest success is that all logic where the two subsystems intersect has now been centralized within the setters of two properties.

While not actually refactoring OSK code in this PR, it's prep work necessary to continue along the lines of the other ✂️ PRs.

User Testing

SUITE_IN_APP: Host app tests.

All behavior exhibited here should feel "normal". These tests are solely to ensure that nothing got accidentally broken.

Platform / keyboard mode combinations

• GROUP_IOS_APP: iOS / Keyman app
• GROUP_ANDROID_APP: Android / System keyboard

Suite tests

• TEST_NORMAL_LOAD: Ensure that the keyboard displays and works correctly when first loaded up.

  Be sure to test one or two keystrokes; ensure that the key produces the expected output.

• TEST_ROTATE_P-TO-L:

  1. With Keyman not active, load it in a portrait orientation.
  2. Once loaded, rotate the device to a landscape orientation.
     • On Android devices, you may need to press something like this for the rotation to occur:
     • The test "passes" if the OSK rotates properly.

• TEST_ROTATE_L-TO-P:

  1. With Keyman not active, load it in a landscape orientation.
  2. Once loaded, rotate the device to a portrait orientation.
     • On Android devices, you may need to press something like this for the rotation to occur:
     • The test "passes" if the OSK rotates properly.

SUITE_DOM: Site-based tests.

All behavior exhibited here should feel "normal". These tests are solely to ensure that nothing got accidentally broken.

OS / Browser combinations

• GROUP_WIN: Windows / Firefox (or Chrome, but just one)
• GROUP_ANDROID_DOM: Android / Chrome
• GROUP_IOS_DOM: iOS (via Simulator) / Safari
  • Note: touches near the bottom of the screen may have some interference from Safari; it likes to display the navigation button bar and "shove" KMW's OSK up a bit. This is expected and not a bug; the relocated OSK should function fine afterward.

For each of these, make use of the "Tests the new Attachment/Enablement API functionality" test page.

Suite tests

• TEST_NORMAL_USE: Ensure that the keyboard displays and works correctly when first loaded up.

  1. Click the "Create Inputs" button one time.
  2. Click / touch the new page element. The OSK should display.
  3. Use the OSK to type one or two keys. The keys should produce their expected output.
  4. Click / touch a blank area of the page. The OSK should automatically hide.

• TEST_ELEMENT_HOPPING:

  1. Click the "Create Inputs" button one time and the "Create Textarea" button one time.
  2. Click / touch the first of the new page elements. The OSK should display.
  3. Click / touch the other page element. The OSK should remain visible.
  4. Click / tough the first element again. The OSK should remain visible.
  5. Click / touch a blank area of the page. The OSK should automatically hide.

• TEST_SPECIFIC_KEYBOARDS:

  1. Click the "Create Inputs" button three times.
  2. Click / touch Dynamic area #0.
  3. Click the element's "Set to Dzongkha" button.
  4. Change the keyboard to French.
  5. Click Dynamic area #1's "Set to Dzongkha button.
  6. Click Dynamic area #1. The Dzongkha keyboard should be displayed.
  7. Click Dynamic area #2.
  8. Change the keyboard to Lao.
  9. Click Dynamic area #0. The French keyboard should be displayed.
  10. Click Dynamic area #2. The Lao keyboard should be displayed.
  11. Click Dynamic area #1. The Dzongkha keyboard should be displayed.
  12. Click Dynamic area #0's "Clear Keyboard" button.
  13. Click Dynamic area #0. The Lao keyboard should be displayed.

SUITE_CJK: Tests CJK keyboard functionality

OS / Browser combinations

• GROUP_MAC: macOS / Firefox

Use the standard "Test unminified KeymanWeb" page.

Suite tests

• TEST_JAPANESE_TYPING: Verify that a CJK keyboard still works.

  Test sequence

  1. Under "Add a keyboard by keyboard name", type `japanese`, then click Add. 2. Select the Japanese keyboard. The OSK should change shape notably: 3. Type `a`. A "picker" displaying a few options should display: 4. Type `2`. The second option should replace the context.

• TEST_JAPANESE_FOCUS: Verify that a CJK keyboard is shown and hidden properly without interfering with a page's UX.

  1. Select the page's textarea element. The OSK's title bar should appear (as with step 2 for the _TYPING test).
  2. Click a blank area on the page. The OSK should automatically hide and the caret should disappear from the textarea.
     • Just in case, "caret": the blinking vertical line segment that lets you know where new characters will appear in the context.
  3. Select the page's input field (the second editable control). The OSK should reappear under the newly-selected control.
  4. Select the page's textarea element again. The OSK should remain visible, and the caret should move from the input control to the textarea control.

---

So, fun fact... the TEST_JAPANESE_FOCUS test above actually fixes an issue that currently exists both on master and on stable_14.0 that had gone unnoticed until this PR's development.

[keymanapp-test-bot[bot]]

User Test Results

✅ SUITE_IN_APP: Host app tests.

• ✅ GROUP_IOS_APP: iOS / Keyman app

  • ✅ TEST_NORMAL_LOAD (PASSED): The keyboard loads and works as expected.
  • ✅ TEST_ROTATE_P-TO-L (PASSED): The keyboard rotates seamlessly. Lovin' it.
  • ✅ TEST_ROTATE_L-TO-P (PASSED): The keyboard rotates seamlessly. Lovin' it.

• ✅ GROUP_ANDROID_APP: Android / System keyboard

  • ✅ TEST_NORMAL_LOAD (PASSED): The keyboard loads without Fatal Error and works as expected.
  • ✅ TEST_ROTATE_P-TO-L (PASSED): No issue found.
  • ✅ TEST_ROTATE_L-TO-P (PASSED): No error after setting Keyman as the system and default keyboard on Android 5.0 and 6.0.

✅ SUITE_DOM: Site-based tests.

• ✅ GROUP_WIN: Windows / Firefox (or Chrome, but just one)

  • ✅ TEST_NORMAL_USE (PASSED): No issue found. Works as intended.
  • ✅ TEST_ELEMENT_HOPPING (PASSED): No issue found.
  • ✅ TEST_SPECIFIC_KEYBOARDS (PASSED): No issue found.

• ✅ GROUP_ANDROID_DOM: Android / Chrome

  • ✅ TEST_NORMAL_USE (PASSED): Similar minor issue with Safari on iOS. See right below.
  • ✅ TEST_ELEMENT_HOPPING (PASSED): The OSK doesn't shown and hide as intended, but the OSK won't appear when clicking in the input area after clicking on the blank area. The culprit might be the same as that of the issue found right above. Since this has nothing to do with hopping, it's marked as PASSED.
  • ✅ TEST_SPECIFIC_KEYBOARDS (PASSED): No issue found.

• ✅ GROUP_IOS_DOM: iOS (via Simulator) / Safari

  • ✅ TEST_NORMAL_USE (PASSED): The OSK does show and hide with a little bit of time between touching the inputs and empty area. (notes)
  • ✅ TEST_ELEMENT_HOPPING (PASSED): No issue found.
  • ✅ TEST_SPECIFIC_KEYBOARDS (PASSED): No issue found.

✅ SUITE_CJK: Tests CJK keyboard functionality

• ✅ GROUP_MAC: macOS / Firefox
  • ✅ TEST_JAPANESE_TYPING (PASSED): The keyboard works as indicated.
  • ✅ TEST_JAPANESE_FOCUS (PASSED) (notes)

[darcywong00]

Can probably reformat TEST_SPECIFIC_KEYBOARDS: so GitHub doesn't link the issue/pull request for #1 and #2.

[MakaraSok]

SUITE_IN_APP: Host app tests.

• GROUP_IOS_APP: iOS / Keyman app

  • TEST_NORMAL_LOAD: PASSED The keyboard loads and works as expected.
  • TEST_ROTATE_P-TO-L: PASSED The keyboard rotates seamlessly. Lovin' it.
  • TEST_ROTATE_L-TO-P: PASSED The keyboard rotates seamlessly. Lovin' it.

• GROUP_ANDROID_APP: Android / System keyboard

  • TEST_NORMAL_LOAD: FAILED A long standing error appears -- Fatal Error.
  

  Note that this error occurs only 5.0.2 right after setting Keyman as system keyboard and default input method.

  On Android 5.0.2, 6.0, 8.1, 10 and 11, there is a fatal error when typing a longpress key on any layer. After that nothing is output from any longpresses, however the normal keys on any layers are working fine after the crash.

  

  Also, there is one weird error which shows up briefly when loading Keyman app in horisontal mode in Android 8.1. It says, "Error in keyboard nul:nul for language".

  
  • TEST_ROTATE_P-TO-L: PASSED The keyboard does retate, but not sure if it is properly. The experience is not as smooth as one expects.
  • TEST_ROTATE_L-TO-P: PASSED The keyboard does retate, but not sure if it is properly. The experience is not as smooth as one expects.

SUITE_DOM: Site-based tests.

• GROUP_WIN: iOS / Keyman app

  • TEST_NORMAL_USE: BLOCKED
  • TEST_ELEMENT_HOPPING: BLOCKED
  • TEST_SPECIFIC_KEYBOARDS: BLOCKED The instructions provided are not applicable for this pair, as Keyman app does not have the Create Inputs button. This case is marked as BLOCKED for now. Kindly guide me through if the steps taken were not expected/intended.

• GROUP_ANDROID_DOM: Android / Chrome

  • TEST_NORMAL_USE: PASSED The OSK appears when touching the input area; OSK hides itself when touching elsewhere.
  

  (I assumed that "new page element" refers to the input area which appears after clicking on the "Create Inputs". If this assumption is wrong, please kindly guide me through as it is detrimental to this test suite.)

  • TEST_ELEMENT_HOPPING: PASSED The OSK does appear, stay and hide itself when touching Textareas, Inputs and away.
  • TEST_SPECIFIC_KEYBOARDS: PASSED Things work as expected.

• GROUP_IOS_DOM: iOS / Safari

  • TEST_NORMAL_USE: BLOCKED Please check the screenshot below.
  

  It looks like the keyboard doesn't load correctly. The keyboard renders unusable. Probably because of my limited knowledge of how to use Safari emulator. This suite is now marked as BLOCKED.

  • TEST_ELEMENT_HOPPING: BLOCKED
  • TEST_SPECIFIC_KEYBOARDS: BLOCKED

SUITE_CJK: Tests CJK keyboard functionality

• GROUP_MAC: macOS / Firefox

  • TEST_JAPANESE_TYPING: PASSED The keyboard works as indicated.
  • TEST_JAPANESE_FOCUS: PASSED

  I blindly assume that "input control" and "textarea control" are the input area on the standard "Test unminified KeymanWeb" page. With that said, the test PASSED. Please guide me if I went off course here.

  Verify that a CJK keyboard is shown and hidden properly without interfering with a page's UX.

  1. Select the page's textarea element. The OSK's title bar should appear (as with step 2 for the _TYPING test).
  2. Click a blank area on the page. The OSK should automatically hide and the caret should disappear from the textarea.
     • Just in case, "caret": the blinking vertical line segment that lets you know where new characters will appear in the context.
  3. Select the page's input field (the second editable control). The OSK should reappear under the newly-selected control.
  4. Select the page's textarea element again. The OSK should remain visible, and the caret should move from the input control to the textarea control.

[jahorton]

@keymanapp/testers

The instructions provided are not applicable for this pair, as Keyman app does not have the Create Inputs button. This case is marked as BLOCKED for now. Kindly guide me through if the steps taken were not expected/intended.

They are applicable for these pairs - you're just referring to the wrong part of the instructions. The DOM tests explicitly say to use the test pages, outside of the apps. Just like you did for the Android one that's part of the same test suite.

[MakaraSok]

@keymanapp/testers

The instructions provided are not applicable for this pair, as Keyman app does not have the Create Inputs button. This case is marked as BLOCKED for now. Kindly guide me through if the steps taken were not expected/intended.

They are applicable for these pairs - you're just referring to the wrong part of the instructions. The DOM tests explicitly say to use the test pages, outside of the apps. Just like you did for the Android one that's part of the same test suite.

Please specify the browser necessary for this test. Should it be in Chrome or FireFox on Windows?

[jahorton]

Oh, I stand corrected for GROUP_WIN. CTRL+C / CTRL+V strikes again; I should have edited that.

I've now changed GROUP_WIN to actually specify Windows things, and the IOS one to specify using iOS Simulator's Safari. You should be able to visit the test pages through Simulator, though this may require logging into TeamCity within the Simulator instance.

[MakaraSok]

SUITE_DOM: Site-based tests.

• GROUP_WIN: Windows / Firefox (or Chrome, but just one)

  • TEST_NORMAL_USE: PASSED
  • TEST_ELEMENT_HOPPING: PASSED
  • TEST_SPECIFIC_KEYBOARDS: PASSED

• GROUP_ANDROID_DOM: Android / Chrome

  • TEST_NORMAL_USE : FAILED The OSK doesn't come about after typing a character, clicking on the blank area of the page and then back in the input area.
  • TEST_ELEMENT_HOPPING: PASSED
  • TEST_SPECIFIC_KEYBOARDS: FAILED For the last step, the OSK stays in French after clicking on "Clear Keyboard"

• GROUP_IOS_DOM: iOS (via Simulator) / Safari

  • TEST_NORMAL_USE: FAILED The OSK hides and doesn't show up again after clicking on the blank area on page.
  • TEST_ELEMENT_HOPPING: PASSED
  • TEST_SPECIFIC_KEYBOARDS: FAILED Tap on the globe key to switch keyboard from Dzongkha to French does not work.

  The keyboard flickered, yet didn't bring up the keyboard menu when trying to switch from Dzongkha to French keyboard. One has to click on it again to see the keyboard menu.

  The last step does not switch the keyboard to Lao. The keyboard stays in French.

[jahorton]

• On Android 5.0.2, 6.0, 8.1, 10 and 11, there is a fatal error when typing a longpress key on any layer. After that nothing is output from any longpresses, however the normal keys on any layers are working fine after the crash.

  

Good catch there - it also would have affected hardware keystrokes, actually. Missed a few spots when changing how the last active element is retrieved.

[jahorton]

The mobile-device bugs (aside from spacebar captions) should now be ready for a retest. I'm also going to add notes for the on-device Safari testing: Safari likes to interfere with language-menu access a bit, as the same location on the screen will display its nav bar.

Spacebar captions are handled separately in #5688.

[jahorton]

Resetting some of the tests:

SUITE_IN_APP GROUP_ANDROID_APP

TEST_NORMAL_LOAD OPEN retest

---

SUITE_DOM OPEN retest

[jahorton]

Okay, so we can't do a full suite that way.

SUITE_DOM

GROUP_ANDROID_DOM OPEN retest GROUP_IOS_DOM OPEN retest

... which doesn't work yet either.

[jahorton]

Well, in that case...

SUITE_DOM

GROUP_WIN

TEST_NORMAL_USE OPEN retest TEST_ELEMENT_HOPPING OPEN retest TEST_SPECIFIC_KEYBOARDS OPEN retest

GROUP_ANDROID_DOM

TEST_NORMAL_USE OPEN retest TEST_ELEMENT_HOPPING OPEN retest TEST_SPECIFIC_KEYBOARDS OPEN retest

GROUP_IOS_DOM

TEST_NORMAL_USE OPEN retest TEST_ELEMENT_HOPPING OPEN retest TEST_SPECIFIC_KEYBOARDS OPEN retest

[MakaraSok]

SUITE_IN_APP: Host app tests.

• GROUP_ANDROID_APP: Android / System keyboard

  • TEST_NORMAL_LOAD: PASSED The keyboard loads without Fatal Error and works as expected.
  • TEST_ROTATE_P-TO-L: PASSED No issue found.
  • TEST_ROTATE_L-TO-P: FAILED Fatal Error in the first attempt to load the keyboard in landscape mode on Android 6.0.
  

  Note that the error doesn't occur again on the next attempt.

  There is also a Fatal Error when typing longpress on the default layer in horizontal mode in Android 8.1. After this error, no longpress outputs anything. Normal press works just fine though.

  

[jahorton]

* **TEST_ROTATE_L-TO-P: FAILED** Fatal Error in the first attempt to load the keyboard in landscape mode on Android 6.0.
  <img alt="" width="70" src="https://user-images.githubusercontent.com/28331388/133953357-21b144ea-ddb5-49ff-8167-53fbb3f566df.png">
  _Note that the error doesn't occur again on the next attempt._
  There is also a Fatal Error when typing longpress on the default layer in horizontal mode in Android 8.1. After this error, no longpress outputs anything. Normal press works just fine though.

I loaded up an emulator with both versions and could reproduce neither issue mentioned here.

[MakaraSok]

SUITE_DOM: Site-based tests.

• GROUP_WIN: Windows / Firefox

  • TEST_NORMAL_USE: PASSED No issue found.
  • TEST_ELEMENT_HOPPING: PASSED No issue found.
  • TEST_SPECIFIC_KEYBOARDS: PASSED No issue found.

• GROUP_ANDROID_DOM: Android / Chrome

  • TEST_NORMAL_USE: FAILED When quickly and alternately click back and forth between the input box and blank area multiple times, the OSK seems to be irresponsive and decided to not showing up.

  Also, after typing a character and then click on the blank area to hide the OSK, the OSK wouldn't be back up when clicking back in the input area.

  • TEST_ELEMENT_HOPPING: PASSED The OSK doesn't shown and hide as intended, but the OSK won't appear when clicking in the input area after clicking on the blank area. The culprit might be the same as that of the issue found right above. Since this has nothing to do with hopping, it's marked as PASSED.
  • TEST_SPECIFIC_KEYBOARDS: PASSED No issue found.

• GROUP_IOS_DOM: iOS (via Simulator) / Safari

  • TEST_NORMAL_USE: PASSED No issue found.
  • TEST_ELEMENT_HOPPING: PASSED No issue found.
  • TEST_SPECIFIC_KEYBOARDS: PASSED No issue found.

[jahorton]

OK, so that last failed user test opened a minor can of worms. There was a "fun" conflict between the focus timer and Android-specific page-scroll handling that simply wasn't ever resolved properly. I think I've figured out how to make the two aspects cooperate, though this did require a few more changes than I'd like. The previous two commits (which accomplished this) may be worth their own review.

As may be noted, I did take the opportunity this provided as an excuse to polish up this area dramatically, removing the old ugly (<any>this.keyman) blocks by providing "proper" definitions where they were previously missing. All changed field and method names were always file- and class-internal, despite where they were being stored. (Things would have been much messier due to the fixes had I not done this.)

[jahorton]

@keymanapp-test-bot retest SUITE_DOM TEST_NORMAL_USE

---

@mcdurdin Looks like that path wasn't quite covered right:

Result of the command above (Seen via drill-down)


Only the first group's test was reset.

So...

@keymanapp-test-bot retest SUITE_DOM GROUP_ANDROID_DOM TEST_NORMAL_USE @keymanapp-test-bot retest SUITE_DOM GROUP_IOS_DOM TEST_NORMAL_USE

---

Okay then...

SUITE_DOM GROUP_ANDROID_DOM TEST_NORMAL_USE OPEN retest

SUITE_DOM GROUP_IOS_DOM TEST_NORMAL_USE OPEN retest

[jahorton]

... maybe it needs an extra comment for this type?

SUITE_DOM GROUP_ANDROID_DOM TEST_NORMAL_USE OPEN retest

SUITE_DOM GROUP_IOS_DOM TEST_NORMAL_USE OPEN retest

Edit: there we go. Finally.

[MakaraSok]

SUITE_DOM: Site-based tests.

• GROUP_ANDROID_DOM: Android / Chrome

  • TEST_NORMAL_USE: PASSED Similar minor issue with Safari on iOS. See right below.

• GROUP_IOS_DOM: iOS (via Simulator) / Safari

  • TEST_NORMAL_USE: PASSED The OSK does show and hide with a little bit of time between touching the inputs and empty area.

  When testing on the emulator, the mouse click may be faster than the reality of when using the actual device. Consequently, the OSK may not behave as intended, unless waiting for a second or so before touching to show or hide it.

  The behavior on web in Windows and macOS is more robust when it comes to show and hide the OSK.

[mcdurdin]

@jahorton you are pushing that poor little test bot too hard. Currently:

1. only one bot command allowed per comment.
2. retest only matches first test, rather than all tests that could match a given string (except in case of 'all' which matches all tests in a suite, group or spec).

So, I expect you could do:

@--keymanapp-test-bot retest SUITE_DOM GROUP_ANDROID_DOM TEST_NORMAL_USE SUITE_DOM GROUP_IOS_DOM TEST_NORMAL_USE

(identifier deliberately mangled to prevent interference.)

Both of these issues should be resolved but as there is a way forward I'll just encourage you to use the recognised incantation for now -- no time this week...

[mcdurdin]

• unless waiting for a second or so

I've also noted these performance issues. See #5729, #5731 for potential mitigations.

[mcdurdin]

The previous two commits (which accomplished this) may be worth their own review.

Okay, re-reviewing. I've got a few things in my queue so hopefully will get to that this morning.

[jahorton]

When testing on the emulator, the mouse click may be faster than the reality of when using the actual device. Consequently, the OSK may not behave as intended, unless waiting for a second or so before touching to show or hide it. The behavior on web in Windows and macOS is more robust when it comes to show and hide the OSK.

Actually, he's referring to something else here:

https://github.com/keymanapp/keyman/blob/3eda9dd3a10071fcf76a3b7ffab6280902358f14/web/source/dom/domEventHandlers.ts#L30-L36

When testing touch-device element attachment, KMW uses this timer to prevent de-focusing selected elements too quickly. I believe that it's been there since before we went open-source.

https://github.com/keymanapp/keyman/blob/a6289f4f0fcc99fb0a29eb8fc97cf0ddbbce458a/web/source/dom/domManager.ts#L1764-L1772

My best guess is that it's there to prevent fat-fingering effects from deselecting an element when the touchpoint is lifted, should the off-element part lift after the on-element part... but this is admittedly conjecture and inference on my part. The block above (and the second comment line) includes fixes to use of this timer, based on said guess.

I agree with @MakaraSok that the timer's probably a bit too long, but I didn't want to make that call in this PR.

[jahorton]

SUITE_IN_APP GROUP_ANDROID_APP TEST_ROTATE_L-TO-P OPEN retest

[MakaraSok]

SUITE_IN_APP: Host app tests.

• GROUP_ANDROID_APP: Android / System keyboard

  • TEST_ROTATE_L-TO-P: FAILED

    1. There is an error after setting the keyboard as system and default keyboard on Android 5.0 and 6.0. It also happens each time the keyboard is loaded in either landscape or portrait mode within the app. This error is not found in Android 8.0, 10.0 and 11.0

    5.0

    

    6.0

    

  2. The loading animation is not as smooth as one would expect it to be. You can see the transition in action while waiting for things to settle.

     6.0

     

SUITE_DOM: Site-based tests.

• GROUP_WIN: Windows / Firefox (or Chrome, but just one)
  • TEST_NORMAL_USE: PASSED No issue found. Works as intended.

[jahorton]

Looks like that bug was addressed on master by https://github.com/keymanapp/keyman/pull/5724. It fits the bill near-perfectly, too.

@keymanapp-test-bot retest SUITE_IN_APP GROUP_ANDROID_APP TEST_ROTATE_L-TO-P retest

[MakaraSok]

SUITE_IN_APP: Host app tests.

• GROUP_ANDROID_APP: Android / System keyboard
  • TEST_ROTATE_L-TO-P: PASSED No error after setting Keyman as the system and default keyboard on Android 5.0 and 6.0.

[keyman-server]

Changes in this pull request will be available for download in Keyman version 15.0.118-alpha