feat(web): Caps Layer and double-tap gesture 🕊

[mcdurdin]

Fixes #3620.

Implements the Caps Lock layer support and the double-tap gesture on the shift key to access it.

The double-tap gesture has been implemented with a view to extension to support other multi-tap gestures in the future. However, for now, it is limited to supporting the Shift key, if and only if the keyboard includes a Caps layer.

The reason for this v15 limitation is that multi-tap on regular keys would involve either rewinding the previous keystroke (the first tap), or forcing keyboard developers to consider 'rota' style rules in their keyboards to support the multi-tap gestures, as we need to make sure that the first tap is accepted and processed for immediate feedback. This needs more design, to avoid unnecessary complexity in the keyboards and/or the rewinding of the keystroke (even though that is conceptually supported in Keyman Engine for Web already). Basically, we don't want to constrain the way that a keyboard author may use the multi-tap gesture by hard-coding the rewind, but neither do we want to make all multi-tap gestures needlessly complex to author.

The shift key (and other modifiers, potentially in future) needs special support for multi-tap as the key that is being tapped changes with the layer change. This is currently managed through recognising K_SHIFT in the key id.

I have tried to follow the PendingGesture pattern for multi-tap, and the gesture itself supports a series of taps, not just a double-tap. The maximum time to complete the tap series is 125msec * number-of-taps, so for a double-tap is 250msec.

The changes to support a Caps Lock layer itself were minimal; just adding the text.KeyboardProcessor.getStateFromLayer function and calling it during KeyEvent construction. The remaining changes relate to the multi-tap gesture.

Minor changes:

• I moved constructNullKeyEvent to KeyEvent in order to make it more accessible to other classes.
• The multi-tap gesture does not have a promise to complete, so that is now an optional member of the PendingGesture interface.

Note:

• The testing page for KeymanWeb includes a test for just the Caps Lock layer support, caps_lock_layer_3620.js, and a full Caps Lock and Start of Sentence test, full_caps_3620_3621.js.

User Testing

SUITE_BASELINE: Test that nothing has changed in existing keyboards

1. Run through basic usage tests of a variety of keyboards, including sil_euro_latin and khmer_angkor.
2. Test longpress keys, layer switching, flick-up gesture.
3. For testing on a physical device: test that rapid touches across multiple keys works correctly, as does multi-touch (where you start tapping the next key before releasing the previous, e.g. rapid two-thumb typing). These are not easy to test in an emulator!

• TEST_BASELINE_WEB: On testing/unminified, run through the above tests. Make sure no script errors are encountered. Do this both for desktop and touch modes.
• TEST_BASELINE_ANDROID: In Keyman for Android, on an Android device or emulator, run through the above tests, both system and in-app keyboards.
• TEST_BASELINE_IPHONE: In Keyman for iPhone, on an iPhone device or emulator, run through the above tests, both system and in-app keyboards.
• TEST_BASELINE_IPAD: In Keyman for iPad, on an iPad device or emulator, run through the above tests, both system and in-app keyboards.

SUITE_CAPS: Test that the new Caps Lock layer controls work correctly

1. Run through a basic usage test of caps_lock_layer_3620.zip.
2. Double-tap the shift key. Watch how it switches to a Caps layer (see the 'shiftlock' style of the Shift key).
3. Test that the Caps layer works correctly. Note that the Shift layer on caps_lock_layer_3620 does not shift back to the base layer automatically; this is intentional.
4. With the full_caps_3620_3621.zip keyboard, test that the Caps layer is accessible through double-tap shift, and that the interactions with Caps Lock and Shift are 'intuitive'.
5. For testing on a physical device: test that rapid touches across multiple keys works correctly, as does multi-touch (where you start tapping the next key before releasing the previous, e.g. rapid two-thumb typing). These are not easy to test in an emulator!

• TEST_CAPS_WEB: On testing/"Test Caps Lock Layer (#3620)", run through the above tests. Make sure no script errors are encountered.
• TEST_CAPS_ANDROID: In Keyman for Android, on an Android device if possible, run through the above tests, both system and in-app keyboards.
• TEST_CAPS_IPHONE: In Keyman for iPhone, on an iPhone device if possible, run through the above tests, both system and in-app keyboards.
• TEST_CAPS_IPAD: In Keyman for iPad, on an iPad device if possible, run through the above tests, both system and in-app keyboards.

[keymanapp-test-bot[bot]]

User Test Results

Test specification and instructions

🟥 SUITE_BASELINE: Test that nothing has changed in existing keyboards

• ✅ TEST_BASELINE_WEB (PASSED): The OSK seems to work fine, but there is a catch. I don't know what "script error" would entail, but as far as the usage of the OSK is concerned, nothing stops me from using the keyboards. (notes)
• 🟥 TEST_BASELINE_ANDROID (FAILED): ???** In Keyman for Android, on an Android device or emulator, run through the above tests, both system and in-app keyboards. (notes)
• ✅ TEST_BASELINE_IPHONE (PASSED): ?** (notes)
• ✅ TEST_BASELINE_IPAD (PASSED): No obvious error is found. (notes)

🟥 SUITE_CAPS: Test that the new Caps Lock layer controls work correctly

• ✅ TEST_CAPS_WEB (PASSED) (notes)
• 🟥 TEST_CAPS_ANDROID (FAILED): ??** (notes)
• 🟥 TEST_CAPS_IPHONE (FAILED) (notes)
• 🟥 TEST_CAPS_IPAD (FAILED) (notes)

Test Artifacts

• Keyman for Android apk
• FirstVoices Keyboards for Android apk
• KMSample1 apk
• KMSample2 apk
• KeymanWeb Test Home
• Keyman for iOS (simulator image)
• FirstVoices Keyboards for iOS (simulator image)
• Keyman for Windows
• Keyman Developer
• FirstVoices Keyboards for Windows

[MakaraSok]

SUITE_BASELINE: Test that nothing has changed in existing keyboards

• TEST_BASELINE_WEB: PASSED The OSK seems to work fine, but there is a catch. I don't know what "script error" would entail, but as far as the usage of the OSK is concerned, nothing stops me from using the keyboards.

  The OSK doesn't show keycaps in capital letters when Caps key is active.

  

  On Touch devices, everything works without any issue, but on iPad: the characters on keycaps are kinda a little too small.

  

• TEST_BASELINE_ANDROID: FAILED?

  • In-app: longpress keys are not working in simulator, but OK in a physical device.

  https://user-images.githubusercontent.com/28331388/145956348-a13b1f93-c616-4639-9a9b-4ec73f30c762.mov

  • System: no obvious issue found, but a glitch found in the rapid touch where the preview key stops showing after some time.

  https://keymanapp.slack.com/files/U6Q7Y6XTM/F02QMS0KDQT/screen_recording_20211214-155720_keyman.mp4

• TEST_BASELINE_IPHONE: PASSED?

  • In-app: No obvious issue found, but there is always a keypress/tap noice when the keyboard finishes starting up. I've never heard this before. Two keypresses/taps are heard in-app and one as a system keyboard. Keyman sometimes crashes (see the beginning of the recording below.)

  https://user-images.githubusercontent.com/28331388/145947540-d938cc36-7d93-4adf-b9ba-78a5db362704.mov

  Note also that the "Get Started" page doesn't show up even though it's on in the settings.

  • System: no issue other than the mysterious keypress/tap noice.

• TEST_BASELINE_IPAD: PASSED No obvious error is found.

  • In-app: Unlike on iPhone, the "Get Started" page is always shown at start up. I've noticed that one keypress/tap noice is heard before the page is shown and another after the page is closed.

  https://user-images.githubusercontent.com/28331388/145950764-baf49665-f570-45ca-8ee2-bff0a098c524.mov

  • System: no issue other than the mysterious keypress/tap noice.

[mcdurdin]

• Make sure no script errors are encountered

To check for script errors when testing KeymanWeb, open the Developer Console (F12 on Chrome/Windows) -- script errors are shown in red and can be located by clicking the button in the top right of the Developer Console.

[mcdurdin]

• System: no obvious issue found, but a glitch found in the rapid touch where the preview key stops showing after some time. https://keymanapp.slack.com/files/U6Q7Y6XTM/F02QMS0KDQT/screen_recording_20211214-155720_keyman.mp4

This video no longer appears available?

[mcdurdin]

• [TEST_BASELINE_ANDROID] In-app: longpress keys are not working in simulator, but OK in a physical device.

This may not be related to this change; can we do a screenshare test together on this?

@keymanapp-test-bot retest SUITE_BASELINE TEST_BASELINE_ANDROID

[MakaraSok]

• System: no obvious issue found, but a glitch found in the rapid touch where the preview key stops showing after some time. https://keymanapp.slack.com/files/U6Q7Y6XTM/F02QMS0KDQT/screen_recording_20211214-155720_keyman.mp4

This video no longer appears available?

https://user-images.githubusercontent.com/28331388/146703733-ae7598e8-b306-4ace-871a-f76fd3df7d64.mp4

[MakaraSok]

SUITE_BASELINE: Test that nothing has changed in existing keyboards

• TEST_BASELINE_ANDROID: FAILED??? In Keyman for Android, on an Android device or emulator, run through the above tests, both system and in-app keyboards.

1. Run through basic usage tests of a variety of keyboards, including sil_euro_latin and khmer_angkor.

OK, but longpress doesn't work on emulator. Nothing like this is seen on a physical device.

2. Test longpress keys, layer switching, flick-up gesture.

OK

3. For testing on a physical device: test that rapid touches across multiple keys works correctly, as does multi-touch (where you start tapping the next key before releasing the previous, e.g. rapid two-thumb typing). These are not easy to test in an emulator!

The key preview may disable during a speed tapping. See the screenshot recording in the comment right above.

[MakaraSok]

SUITE_CAPS: Test that the new Caps Lock layer controls work correctly

• TEST_CAPS_ANDROID: FAILED??

1. Run through a basic usage test of caps_lock_layer_3620.zip.

One of the basic usage of the keyboard doesn't work as expected. Long press keys are not responsive; the subkeys do not pop up as expected on Android 10 simulator API 29. Note that this doesn't happen on the physical device.

2. Double-tap the shift key. Watch how it switches to a Caps layer (see the 'shiftlock' style of the Shift key).

Working OK.

3. Test that the Caps layer works correctly. Note that the Shift layer on caps_lock_layer_3620 does not shift back to the base layer automatically; this is intentional.

Each key is working OK, but those keys with longpresses on the simulator. On the physical device however, this doesn't happen.

4. With the full_caps_3620_3621.zip keyboard, test that the Caps layer is accessible through double-tap shift, and that the interactions with Caps Lock and Shift are 'intuitive'.

Each key is working OK, but those keys with longpresses. On the physical device however, this doesn't happen.

5. For testing on a physical device: test that rapid touches across multiple keys works correctly, as does multi-touch (where you start tapping the next key before releasing the previous, e.g. rapid two-thumb typing). These are not easy to test in an emulator!

The test is not quite practical, but it does appear that the preview went off and on amid typing.

https://user-images.githubusercontent.com/28331388/146709335-7343e4c0-dc42-419d-b042-48551177f31a.mp4

[MakaraSok]

SUITE_CAPS: Test that the new Caps Lock layer controls work correctly

• TEST_CAPS_IPHONE: FAILED

1. Run through a basic usage test of caps_lock_layer_3620.zip.

Nothing weird observed.

2. Double-tap the shift key. Watch how it switches to a Caps layer (see the 'shiftlock' style of the Shift key).

Working OK, but the weird thing is the double tap seem unnatural to the touch. You don't hear the two taps, just one is heard. :/

3. Test that the Caps layer works correctly. Note that the Shift layer on caps_lock_layer_3620 does not shift back to the base layer automatically; this is intentional.

Each key is working OK including longpresses.

4. With the full_caps_3620_3621.zip keyboard, test that the Caps layer is accessible through double-tap shift, and that the interactions with Caps Lock and Shift are 'intuitive'.

Not only that the double tap is unnatural to the touch, it doesn't enable caps layer at all. Double tap the Shift key and start typing SS gives you Ss instead.

5. For testing on a physical device: test that rapid touches across multiple keys works correctly, as does multi-touch (where you start tapping the next key before releasing the previous, e.g. rapid two-thumb typing). These are not easy to test in an emulator!

No physical device within reach right now.

[MakaraSok]

SUITE_CAPS: Test that the new Caps Lock layer controls work correctly

• TEST_CAPS_IPAD: FAILED

1. Run through a basic usage test of caps_lock_layer_3620.zip.

Nothing weird observed.

2. Double-tap the shift key. Watch how it switches to a Caps layer (see the 'shiftlock' style of the Shift key).

Working OK, but the weird thing is the double tap seem unnatural to the touch. You don't hear the two taps, just one is heard. :/

3. Test that the Caps layer works correctly. Note that the Shift layer on caps_lock_layer_3620 does not shift back to the base layer automatically; this is intentional.

Each key is working OK including longpresses.

4. With the full_caps_3620_3621.zip keyboard, test that the Caps layer is accessible through double-tap shift, and that the interactions with Caps Lock and Shift are 'intuitive'.

Not only that the double tap is unnatural to the touch, it doesn't enable caps layer at all. Double tap the Shift key and start typing SS gives you Ss instead.

https://user-images.githubusercontent.com/28331388/146721426-8d558d95-101d-461f-9477-322b12a1e7b0.mov

5. For testing on a physical device: test that rapid touches across multiple keys works correctly, as does multi-touch (where you start tapping the next key before releasing the previous, e.g. rapid two-thumb typing). These are not easy to test in an emulator!

No physical device within reach right now.

[MakaraSok]

SUITE_CAPS: Test that the new Caps Lock layer controls work correctly

• TEST_CAPS_WEB: PASSED

1. Run through a basic usage test of caps_lock_layer_3620.zip.

Nothing weird observed.

2. Double-tap the shift key. Watch how it switches to a Caps layer (see the 'shiftlock' style of the Shift key).

Working OK.

3. Test that the Caps layer works correctly. Note that the Shift layer on caps_lock_layer_3620 does not shift back to the base layer automatically; this is intentional.

Each key is working OK including longpresses.

4. With the full_caps_3620_3621.zip keyboard, test that the Caps layer is accessible through double-tap shift, and that the interactions with Caps Lock and Shift are 'intuitive'.

OK

5. For testing on a physical device: test that rapid touches across multiple keys works correctly, as does multi-touch (where you start tapping the next key before releasing the previous, e.g. rapid two-thumb typing). These are not easy to test in an emulator!

No physical device within reach right now.

[mcdurdin]

Failing test feedback:

• Android: the longpress issue on emulator is captured at #6113.
• iOS: the issue with in-app context loss causing the caps lock layer to cancel is probably the same as #6112.
• iOS: the issue with double-tap not giving a double click is now captured in #6121.

As I have now captured the three failure cases as separate issues, I will go ahead and merge this PR, with the caveat that these issues must be resolved before release.

[keyman-server]

Changes in this pull request will be available for download in Keyman version 15.0.180-alpha

[MayuraVerma]

How to enable CAPS LOCK with double tap in Keyboard?

[MayuraVerma]

I have tried in iOS 15.0.188-alpha with EUROSIL no sign of caps lock with double tap on shift key.

[MayuraVerma]

I tested full_caps_3620_3621.zip with iOS 15.0.188-alpha

CAPS lock turns off, after a key stroke CAPS lock should stay in that state until turned off if not there is not difference between CAPS lock and shift layer.

[mcdurdin]

@MakaraSok or @bharanidharanj, are you able to replicate the issues that @MayuraVerma has experienced? Can you work with Mayura to build a reproducible test case so we can resolve this before release?

[mcdurdin]

I have tried in iOS 15.0.188-alpha with EUROSIL

The SIL Eurolatin keyboard has not yet been updated to support Caps Lock 😄

[MayuraVerma]

I have tried in iOS 15.0.188-alpha with EUROSIL

The SIL Eurolatin keyboard has not yet been updated to support Caps Lock 😄

Yes, I realized it later. since then I tested full_caps_3620_3621.zip with iOS 15.0.188-alpha

Capslock works. But I am thinking we are leaving it to the keyboard developer to choose the next layer after output in CAPS layer, default should be it should remain in CAPS layer. Once we enter CAPS layer, only why to get out of CAPS layer should be release the CAPS lock. This is how it works in hardware, we should mimic the same in software keyboard also.

[MayuraVerma]

I have tried in iOS 15.0.188-alpha with EUROSIL

The SIL Eurolatin keyboard has not yet been updated to support Caps Lock 😄

My earlier assumption was, web/ios/android picks up the NCAPS state from kmn. looks like we need to rewrite it mobile version.

[mcdurdin]

My earlier assumption was, web/ios/android picks up the NCAPS state from kmn.

Yeah, unfortunately given how the touch and physical keyboards diverge, we could not find a safe way to do this automatically. My preference is always to give the control to the keyboard designer, in any case -- but the trade-off is that we don't get this kind of behaviour "for free".

[mcdurdin]

Once we enter CAPS layer, only why to get out of CAPS layer should be release the CAPS lock.

There are two ways, in our design, that the Caps Lock layer is released on touch:

1. When the user touches the Caps Lock key again (or another modifier key accessible on the Caps Lock layer).
2. When the user switches apps or input contexts. This matches the behaviour of the system keyboards -- Caps Lock state is not maintained long-term, unlike on a hardware keyboard.

So if you are seeing behaviour that does not match those conditions, that is definitely a bug. And if you can give repro steps, that will help us squash the bug!