MakaraSok commented 1 year ago

Keyman for iPhone and iPad Acceptance Test Procedures

These test procedures are to be run before moving from alpha to beta, or beta to stable, or before PRs are merged into stable branches.
Copy these tests into a new issue (for tier transitions) or a comment on the PR.
When annotating results, use > at the start of a new line under the checkbox to note the result. (For an example, see #3973).

User Testing

Gather Assets for Testing

[ ] Record device's iOS version
[ ] Ensure the device is properly configured for testing and is registered with TestFlight.

At this time, Keyman for iOS does not support external keyboard input.

Setup Steps

Uninstall previous version of Keyman on the device (on a simulator, "Erase All Content and Settings..." under Device menu)
Install test build IPA (on a simulator, "iOS Simulator Image")
Disable internet connectivity of the device
Start Keyman and select Add a keyboard for your language
Verify error message "Could not reach Keyman server!" or "The Internet connection appears to be offline." is displayed
Click OK to dismiss. Close Keyman app
On device, enable internet connectivity
Start Keyman
On initial Get Started menu, select "Enable Keyman as system-wide keyboard" and do any necessary system steps to enable Keyman

SUITE_INSPECT: Visual Inspection of master branch

TEST_BUILD_STATUS: Verify latest CI build of master is successful
TEST_VERIFY_HISTORY: Verify HISTORY.md contains all the current changes
TEST_VERIFY_UNIT_TESTS: Verify that all unit tests for iOS are currently passing.
- May be integrated with the first step above once our usual CI processes include the iOS unit tests.

SUITE_MENU_FUNCTIONALITY: Menu Functionality Tests, in-app

GROUP_IPHONE_PORTRAIT: portrait mode in iPhone
GROUP_IPHONE_LANDSCAPE: landscape mode in iPhone
GROUP_IPAD_PORTRAIT: portrait mode in iPad
GROUP_IPAD_LANDSCAPE: landscape mode in iPad

Test Cases

TEST_GET_STARTED: Get Started
- Verify touching Add a keyboard for your language pulls up the keyboard search
  - This should also be accessible as Settings > Installed Languages > +.
- Verify touching Enable Keyman as system-wide keyboard pulls up Keyman's menu within iOS settings. (11.0+ devices)
  - If older versions of iOS are supported, ensure that the help page for doing it manually is properly linked.
- Verify touching More info brings up an accurate version and the main table-of-contents help page.
  - This should also be accessible as Info from the main menu dropdown (on phones) or as the circled-i button (on tablets).
- Set Don't show again to checked, close and re-open Keyman, and verify Get Started menu does not appear
TEST_ADD_NEW_KEYBOARD: Add New Keyboard This menu is accessed via Get Started menu or Settings menu
- On the device, enable internet connectivity
- Verify a new keyboard can be downloaded and selected, say "US Basic"
- Return to the text input screen and confirm that it operates correctly.
- Without changing selected language, confirm that it can be cleanly deleted via the Settings menu.
- Confirm that another keyboard is correctly selected afterward.
TEST_ADD_NEW_KEYBOARD_FROM_FILE:
- On the device, go to keyman.com on Safari
- Search for "US Basic"
- scroll down to keyboard details and tap to download the keyboard package (i.e. .kmp file)
- open Files app and navigate to Download folder to find "basic_kbdus.kmp" and tap on it to start the installation processes
- Keyman app should be pulled up
- follow the screen to finish the keyboard installation
- verity that the keyboard is correctly installed and working as expected
TEST_SWITCH_KEYBOARD
- press on the globe key
- the keyboard menu should be shown
- press on a keyboard to switch to, say "EuroLatin (SIL)"
- confirm that "EuroLatin (SIL)" is now selected and ready to be used
- swich back to "US Basic" and confirm it does as expected
TEST_SHARE: Share
- Type and verify text can be copied and shared to external app
TEST_KEYMAN_BROWSER: Keyman Browser ###
- On default page, click on text field and set system keyboard to Keyman
- Verify can type with Keyman as system keyboard
- Verify that predictive suggestions are properly applied when selected (using sil_euro_latin + English)
- Close and reopen Keyman app
TEST_TEXT_SIZE: Text Size
- Verify text can be rescaled from Text Size 9 to 72
TEST_CLEAR_TEXT: Clear Text
- Verify text can be cleared
TEST_INFO: Info
- Verify that the Keyman iOS version is displayed; record the version
- Verify that the Info menu options matches that of the More Info... option from the Get Started menu option.
- Verify that the Keyman iOS version matches the version string seen at (Home Screen) > [iOS] Settings > Keyman.
TEST_SETTINGS: Settings
- Installed Languages ... Add Language
- Select an installed language ... Add (another) Keyboard
- Should automatically perform a filtered search based upon that language's BCP-47 code, i.e. for "English" it should be seen as shown in the screenshot below.
- Select an installed language ... Dictionary
- Turn on/off 'Enable Predictions' and validate each
- Turn on/off 'Enable Corrections' and validate each
- Validate that when both predictions and corrections are off, banner is not visible
- Add/Remove dictionaries - validate
- If multiple dictionaries are available, test swapping between them
TEST_EUROLATIN: English (Eurolatin) in-app
- In portrait orientation, verify that the OSK appears and fills the width the bottom of the screen
- Verify long-press q key works
- Verify long-press k key works
  - Verify that any predictive suggestions generated feel 'reasonable' (based upon 'k', the selected subkey, or one of its neighbors)
- Verify uppercase layer can be selected via SHIFT
- Verify number layer can be selected via 123
- Verify long-press 1 key works
- Verify long-press 0 key works
- Verify backspace, space, and enter keys work

Note that EuroLatin2, a highly customized keyboard explicitly designed for mobile, is not found; hence EuroLatin (SIL) is used for testing this instead.

TEST_US_BASIC: English: US Basic in-app
- Verify that the US Basic keyboard can be downloaded through Settings > Installed Languages > English > +
- This should automatically bring up search results for English keyboards.
- When the package-installer launches, English should be the one pre-selected language.
- Install the package for both English and Tagalog by selecting both of them in the installation processes.
- Set English: US Basic as the active in-app keyboard.
- Verify that predictive text suggestions appear.
- Set Tagalog: US Basic as the active in-app keyboard.
- Verify that predictive text suggestions do not appear.
- Verify that long-press 123 key works, presenting options for a currency layer and a symbol layer.
- Select the currency layer, then press the £ key.
- Verify that the keyboard outputs correctly and returns to the default layer.
- Go to the symbol layer and press the © key.
- Verify that the keyboard outputs correctly and remains on the symbol layer.
- Revert to the default layer and ensure basic key inputs work.
- Long-press e on the default layer and select a subkey.
- Verify that the selected key produces the correct output.
- From the Settings > Languages > English > US Basic menu, follow the "Help link" and ensure it displays appropriate help.
- (Should work both online and offline)
- From the Settings > Languages > English > US Basic menu, scan the QR code with a phone and test that it links to the current version of that keyboard's public download page on keyman.com.
- Returning to the app, uninstall the US Basic keyboard via Settings for Tagalog.
- Set English: US Basic as the active in-app keyboard.
- Verify that the keyboard appears correctly and ensure basic key inputs work.
- This is to ensure that uninstalling for one language doesn't adversely affect installations of the same resource for another language.
- Uninstall the US Basic keyboard via Settings for English.

SUITE_SYSTEM_KEYBOARD:

GROUP_IPHONE_PORTRAIT: portrait mode in iPhone
GROUP_IPHONE_LANDSCAPE: landscape mode in iPhone
GROUP_IPAD_PORTRAIT: portrait mode in iPad
GROUP_IPAD_LANDSCAPE: landscape mode in iPad

Test the following after setting Keyman as a system keyboard.

Test Cases

TEST_BASIC_USAGE: ensure that the keyboards functions as expected on all layers. Please test on multiple keyboards, i.e. EuroLatin (SIL), Cameroon QWERTY keyboard and Khmer Angkor.
TEST_LONG_PRESS: ensure that longpress on all applicable layers are working as intended
TEST_GLOBE_KEY: ensure that a short press on the globe key switch to the next keyboard and a longpress on it open the keyboard menu where you can select a keyboard of your choice

SUITE_SWITCH_LOCALE

GROUP_IPHONE_PORTRAIT: portrait mode in iPhone
GROUP_IPHONE_LANDSCAPE: landscape mode in iPhone
GROUP_IPAD_PORTRAIT: portrait mode in iPad
GROUP_IPAD_LANDSCAPE: landscape mode in iPad

Test Cases

TEST_KM_LOCALE
- from System Preferences, switch the UI language to Khmer by adding it in Language & Region using +
- drag Khmer to the top of the list to set it as a primary UI language
- check the dropdown menu to see it in Khmer (there is a known issue on this though, cf. https://github.com/keymanapp/keyman/issues/6527)
- verify the UI language switch to Khmer
TEST_AM_LOCALE
- from System Preferences, switch the UI language to Amharic by adding it in Language & Region using +
- drag Amharic to the top of the list to set it as a primary UI language
- check the dropdown menu to see it in Amharic (there is a known issue on this though, cf. https://github.com/keymanapp/keyman/issues/6527)
- verify the UI language switch to Amharic

[TODO] SUITE_PREDICTIVE_TEXT: Predictive Text

keymanapp-test-bot[bot] commented 1 year ago

User Test Results

Test specification and instructions

✅ SUITE_INSPECT: Visual Inspection of master branch

3 tests in 1 groups PASSED

- ✅ **TEST_BUILD_STATUS ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1318640356))**: Verified the latest master branch beta build is successful. ([notes](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1318640356)) - ✅ **TEST_VERIFY_HISTORY ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1319720140))**: Retested this with the updated link and it is opening the correct page. ([notes](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1319720140)) - ✅ **TEST_VERIFY_UNIT_TESTS ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324867023))**: Tested with Keyman 16.0.108-beta build in all the areas in iPhone and iPad Simulators and all the tests are passed. Thanks.

✅ SUITE_MENU_FUNCTIONALITY: Menu Functionality Tests, in-app

48 tests in 4 groups PASSED

* ✅ GROUP_IPHONE_PORTRAIT: portrait mode in iPhone

12 tests PASSED

- ✅ **TEST_GET_STARTED ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1321961075))**: Tested with Keyman 16.0.107-beta build in iPhone 12 Pro Max Simulator and the Get Started functionality is working as expected. - ✅ **TEST_ADD_NEW_KEYBOARD ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1321961075))**: Verified that we are able to install a new keyboard as per the instruction. - ✅ **TEST_ADD_NEW_KEYBOARD_FROM_FILE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1321961075))**: Verified that we able to install a new keyboard from file. - ✅ **TEST_SWITCH_KEYBOARD ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1321961075))**: Able to switch between keyboards using the Keyboard picker menu. - ✅ **TEST_SHARE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1321961075))**: Able to share the text with some other applications like Message App. - ✅ **TEST_KEYMAN_BROWSER ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1321961075))**: Verified that the predictive suggestions are properly applied when selected English - EuroLatin SIL keyboard. - ✅ **TEST_TEXT_SIZE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1321961075))**: Verified that I am able to rescale the text size from 9 to 72. - ✅ **TEST_CLEAR_TEXT ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1321961075))**: Able to clear using Clear Text option. - ✅ **TEST_INFO ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1321961075))**: Verfied the Keyman Info option from Get Started menu and in the Home Screen. - ✅ **TEST_SETTINGS ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1321961075))**: Verified the Keyman Settings functionality and it is working as expected. - ✅ **TEST_EUROLATIN ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1321961075))**: Verified the EuroLatin Keyman and it showing the expected results. - ✅ **TEST_US_BASIC ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1321961075))**: Verified the English: US Basic as well as the Tagalog: US Basic and both are working as expected.

* ✅ GROUP_IPHONE_LANDSCAPE: landscape mode in iPhone

12 tests PASSED

- ✅ **TEST_GET_STARTED ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322136598))**: Tested with Keyman 16.0.107-beta build in iPhone 12 Pro Max Simulator and the Get Started functionality is working as expected. - ✅ **TEST_ADD_NEW_KEYBOARD ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322136598))**: Verified that we are able to install a new keyboard as per the instruction. - ✅ **TEST_ADD_NEW_KEYBOARD_FROM_FILE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322136598))**: Verified that we able to install a new keyboard from file. - ✅ **TEST_SWITCH_KEYBOARD ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322136598))**: Able to switch between keyboards using the Keyboard picker menu. - ✅ **TEST_SHARE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322136598))**: Able to share the text with some other applications like Message App. - ✅ **TEST_KEYMAN_BROWSER ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322136598))**: Verified that the predictive suggestions are properly applied when selected English - EuroLatin SIL keyboard. - ✅ **TEST_TEXT_SIZE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322136598))**: Verified that I am able to rescale the text size from 9 to 72. - ✅ **TEST_CLEAR_TEXT ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322136598))**: Able to clear using Clear Text option. - ✅ **TEST_INFO ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322136598))**: Verfied the Keyman Info option from Get Started menu and in the Home Screen. - ✅ **TEST_SETTINGS ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322136598))**: Verified the Keyman Settings functionality and it is working as expected. - ✅ **TEST_EUROLATIN ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322136598))**: Verified the EuroLatin Keyman and it showing the expected results. - ✅ **TEST_US_BASIC ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322136598))**: Verified the English: US Basic as well as the Tagalog: US Basic and both are working as expected.

* ✅ GROUP_IPAD_PORTRAIT: portrait mode in iPad

12 tests PASSED

- ✅ **TEST_GET_STARTED ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322192895))**: Tested with Keyman 16.0.107-beta build in iOS 15.0 / iPad Air Simulator and the Get Started functionality is working as expected. - ✅ **TEST_ADD_NEW_KEYBOARD ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322192895))**: Verified that we are able to install a new keyboard as per the instruction. - ✅ **TEST_ADD_NEW_KEYBOARD_FROM_FILE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322192895))**: Verified that we able to install a new keyboard from file. - ✅ **TEST_SWITCH_KEYBOARD ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322192895))**: Able to switch between keyboards using the Keyboard picker menu. - ✅ **TEST_SHARE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322192895))**: Able to share the text with some other applications like Message App. - ✅ **TEST_KEYMAN_BROWSER ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322192895))**: Verified that the predictive suggestions are properly applied when selected English - EuroLatin SIL keyboard. - ✅ **TEST_TEXT_SIZE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322192895))**: Verified that I am able to rescale the text size from 9 to 72. - ✅ **TEST_CLEAR_TEXT ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322192895))**: Able to clear using Clear Text option. - ✅ **TEST_INFO ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322192895))**: Verified the Keyman Info option from Get Started menu and in the Home Screen. - ✅ **TEST_SETTINGS ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322192895))**: Verified the Keyman Settings functionality and it is working as expected. - ✅ **TEST_EUROLATIN ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322192895))**: Verified the EuroLatin Keyman and it showing the expected results. - ✅ **TEST_US_BASIC ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322192895))**: Verified the English: US Basic as well as the Tagalog: US Basic and both are working as expected.

* ✅ GROUP_IPAD_LANDSCAPE: landscape mode in iPad

12 tests PASSED

- ✅ **TEST_GET_STARTED ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322204317))**: Tested with Keyman 16.0.107-beta build in iOS 15.0 / iPad Air Simulator and the Get Started functionality is working as expected. - ✅ **TEST_ADD_NEW_KEYBOARD ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322204317))**: Verified that we are able to install a new keyboard as per the instruction. - ✅ **TEST_ADD_NEW_KEYBOARD_FROM_FILE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322204317))**: Verified that we able to install a new keyboard from file. - ✅ **TEST_SWITCH_KEYBOARD ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322204317))**: Able to switch between keyboards using the Keyboard picker menu. - ✅ **TEST_SHARE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322204317))**: Able to share the text with some other applications like Message App. - ✅ **TEST_KEYMAN_BROWSER ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322204317))**: Verified that the predictive suggestions are properly applied when selected English - EuroLatin SIL keyboard. - ✅ **TEST_TEXT_SIZE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322204317))**: Verified that I am able to rescale the text size from 9 to 72. - ✅ **TEST_CLEAR_TEXT ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322204317))**: Able to clear using Clear Text option. - ✅ **TEST_INFO ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322204317))**: Verified the Keyman Info option from Get Started menu and in the Home Screen. - ✅ **TEST_SETTINGS ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322204317))**: Verified the Keyman Settings functionality and it is working as expected. - ✅ **TEST_EUROLATIN ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322204317))**: Verified the EuroLatin Keyman and it showing the expected results. - ✅ **TEST_US_BASIC ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1322204317))**: Verified the English: US Basic as well as the Tagalog: US Basic and both are working as expected.

✅ SUITE_SYSTEM_KEYBOARD:

12 tests in 4 groups PASSED

* ✅ GROUP_IPHONE_PORTRAIT: portrait mode in iPhone

3 tests PASSED

- ✅ **TEST_BASIC_USAGE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1323624997))**: Tested this as per the instructions with some of the keyman keyboards (eg., khmer_angkor, cameroon_qwerty, tamilzha 99 etc.,) and I was able to use it in all layers of the keyboard. - ✅ **TEST_LONG_PRESS ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1323624997))**: Verified that long press on all applicable layers are working as expected. - ✅ **TEST_GLOBE_KEY ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324645306))**: Retested this as per Shawn's suggestion and now I am able to see the Keyboard picker menu while long pressing the key in a browser. ([notes](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324645306))

* ✅ GROUP_IPHONE_LANDSCAPE: landscape mode in iPhone

3 tests PASSED

- ✅ **TEST_BASIC_USAGE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1323624997))**: Tested this as per the instructions with some of the keyman keyboards (eg., khmer_angkor, cameroon_qwerty, tamilzha 99 etc.,) and I was able to use it in all layers of the keyboard. - ✅ **TEST_LONG_PRESS ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1323624997))**: Verified that long press on all applicable layers are working as expected. - ✅ **TEST_GLOBE_KEY ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324645306))**: Retested this as per Shawn's suggestion and now I am able to see the Keyboard picker menu while long pressing the key in a browser. ([notes](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324645306))

* ✅ GROUP_IPAD_PORTRAIT: portrait mode in iPad

3 tests PASSED

- ✅ **TEST_BASIC_USAGE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324784912))**: Tested with Keyman 16.108-beta build in iPad Air (4th generation) / iOS 15.0 Simulator and I was able to use all the layers of the keyboards (khmer_angkor, sil_eurolatin and cameroon qwerty) without having any issues. - ✅ **TEST_LONG_PRESS ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324784912))**: Verified that long press on all applicable layers are working as expected. - ✅ **TEST_GLOBE_KEY ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324784912))**: After changing Keyman as the System wide keyboard, I was able to see the Keyboard picker menu while long pressing the key in a browser. ([notes](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324784912))

* ✅ GROUP_IPAD_LANDSCAPE: landscape mode in iPad

3 tests PASSED

- ✅ **TEST_BASIC_USAGE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324784912))**: Verified in all layers of the keyboards (khmer_angkor, sil_eurolatin and cameroon qwerty) without having any issues. - ✅ **TEST_LONG_PRESS ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324784912))**: Verified that long press on all applicable layers are working as expected. - ✅ **TEST_GLOBE_KEY ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324784912))**: After changing Keyman as the System wide keyboard, I was able to see the Keyboard picker menu while long pressing the key in a browser. ([notes](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324784912))

✅ SUITE_SWITCH_LOCALE:

8 tests in 4 groups PASSED

* ✅ GROUP_IPHONE_PORTRAIT: portrait mode in iPhone

2 tests PASSED

- ✅ **TEST_KM_LOCALE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1323646929))**: Verified that the UI language has been changed to Khmer in Keyman App. ([notes](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1323646929)) - ✅ **TEST_AM_LOCALE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1323646929))**: Verified that the UI language has been changed to Amharic in Keyman App. ([notes](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1323646929))

* ✅ GROUP_IPHONE_LANDSCAPE: landscape mode in iPhone

2 tests PASSED

- ✅ **TEST_KM_LOCALE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1323646929))**: Verified that the UI language has been changed to Khmer in Keyman App. ([notes](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1323646929)) - ✅ **TEST_AM_LOCALE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1323646929))**: Verified that the UI language has been changed to Amharic in Keyman App. ([notes](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1323646929))

* ✅ GROUP_IPAD_PORTRAIT: portrait mode in iPad

2 tests PASSED

- ✅ **TEST_KM_LOCALE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324792853))**: Verified that the UI language has been changed to Khmer in Keyman App. ([notes](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324792853)) - ✅ **TEST_AM_LOCALE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324792853))**: Verified that the UI language has been changed to Amharic in Keyman App.

* ✅ GROUP_IPAD_LANDSCAPE: landscape mode in iPad

2 tests PASSED

- ✅ **TEST_KM_LOCALE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324792853))**: Verified that the UI language has been changed to Khmer in Keyman App. - ✅ **TEST_AM_LOCALE ([PASSED](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324792853))**: Verified that the UI language has been changed to Amharic in Keyman App. ([notes](https://github.com/keymanapp/keyman/issues/7618#issuecomment-1324792853))

bharanidharanj commented 1 year ago

SUITE_INSPECT: Visual Inspection of master branch

TEST_BUILD_STATUS (PASSED): Verified the latest master branch beta build is successful.
TEST_VERIFY_HISTORY (FAILED): History File need to be updated for iOS.

mcdurdin commented 1 year ago

TEST_VERIFY_HISTORY: Verify HISTORY.md contains all the current changes

This test spec is incorrect -- the link should be to https://github.com/keymanapp/keyman/tree/beta/HISTORY.md

@keymanapp-test-bot retest TEST_VERIFY_HISTORY

bharanidharanj commented 1 year ago

SUITE_INSPECT: Visual Inspection of master branch

TEST_VERIFY_HISTORY (PASSED): Retested this with the updated link and it is opening the correct page.

bharanidharanj commented 1 year ago

SUITE_MENU_FUNCTIONALITY: Menu Functionality Tests, in-app

GROUP_IPHONE_PORTRAIT: portrait mode in iPhone

TEST_GET_STARTED (PASSED): Tested with Keyman 16.0.106-beta build in iPhone 12 Pro Max Simulator and the Get Started functionality is working as expected.
TEST_ADD_NEW_KEYBOARD (PASSED): Verified that we are able to install a new keyboard as per the instruction.
TEST_ADD_NEW_KEYBOARD_FROM_FILE (PASSED): Verified that we able to install a new keyboard from file. (Download folder. )
TEST_SWITCH_KEYBOARD (PASSED): Able to switch between keyboards using the Keyboard picker menu.
TEST_SHARE (PASSED): Able to share the text with some other applications like Message App.

bharanidharanj commented 1 year ago

SUITE_MENU_FUNCTIONALITY: Menu Functionality Tests, in-app

GROUP_IPHONE_PORTRAIT: portrait mode in iPhone

TEST_GET_STARTED (PASSED): Tested with Keyman 16.0.107-beta build in iPhone 12 Pro Max Simulator and the Get Started functionality is working as expected.
TEST_ADD_NEW_KEYBOARD (PASSED): Verified that we are able to install a new keyboard as per the instruction.
TEST_ADD_NEW_KEYBOARD_FROM_FILE (PASSED): Verified that we able to install a new keyboard from file.
TEST_SWITCH_KEYBOARD (PASSED): Able to switch between keyboards using the Keyboard picker menu.
TEST_SHARE (PASSED): Able to share the text with some other applications like Message App.
TEST_KEYMAN_BROWSER (PASSED): Verified that the predictive suggestions are properly applied when selected English - EuroLatin SIL keyboard.
TEST_TEXT_SIZE (PASSED): Verified that I am able to rescale the text size from 9 to 72.
TEST_CLEAR_TEXT (PASSED): Able to clear using Clear Text option.
TEST_INFO (PASSED): Verfied the Keyman Info option from Get Started menu and in the Home Screen.
TEST_SETTINGS (PASSED): Verified the Keyman Settings functionality and it is working as expected.
TEST_EUROLATIN (PASSED): Verified the EuroLatin Keyman and it showing the expected results.
TEST_US_BASIC (PASSED): Verified the English: US Basic as well as the Tagalog: US Basic and both are working as expected.

bharanidharanj commented 1 year ago

SUITE_MENU_FUNCTIONALITY: Menu Functionality Tests, in-app

GROUP_IPHONE_LANDSCAPE: landscape mode in iPhone

TEST_GET_STARTED (PASSED): Tested with Keyman 16.0.107-beta build in iPhone 12 Pro Max Simulator and the Get Started functionality is working as expected.
TEST_ADD_NEW_KEYBOARD (PASSED): Verified that we are able to install a new keyboard as per the instruction.
TEST_ADD_NEW_KEYBOARD_FROM_FILE (PASSED): Verified that we able to install a new keyboard from file.
TEST_SWITCH_KEYBOARD (PASSED): Able to switch between keyboards using the Keyboard picker menu.
TEST_SHARE (PASSED): Able to share the text with some other applications like Message App.
TEST_KEYMAN_BROWSER (PASSED): Verified that the predictive suggestions are properly applied when selected English - EuroLatin SIL keyboard.
TEST_TEXT_SIZE (PASSED): Verified that I am able to rescale the text size from 9 to 72.
TEST_CLEAR_TEXT (PASSED): Able to clear using Clear Text option.
TEST_INFO (PASSED): Verfied the Keyman Info option from Get Started menu and in the Home Screen.
TEST_SETTINGS (PASSED): Verified the Keyman Settings functionality and it is working as expected.
TEST_EUROLATIN (PASSED): Verified the EuroLatin Keyman and it showing the expected results.
TEST_US_BASIC (PASSED): Verified the English: US Basic as well as the Tagalog: US Basic and both are working as expected.

bharanidharanj commented 1 year ago

SUITE_MENU_FUNCTIONALITY: Menu Functionality Tests, in-app

GROUP_IPAD_PORTRAIT: portrait mode in iPad

TEST_GET_STARTED (PASSED): Tested with Keyman 16.0.107-beta build in iOS 15.0 / iPad Air Simulator and the Get Started functionality is working as expected.
TEST_ADD_NEW_KEYBOARD (PASSED): Verified that we are able to install a new keyboard as per the instruction.
TEST_ADD_NEW_KEYBOARD_FROM_FILE (PASSED): Verified that we able to install a new keyboard from file.
TEST_SWITCH_KEYBOARD (PASSED): Able to switch between keyboards using the Keyboard picker menu.
TEST_SHARE (PASSED): Able to share the text with some other applications like Message App.
TEST_KEYMAN_BROWSER (PASSED): Verified that the predictive suggestions are properly applied when selected English - EuroLatin SIL keyboard.
TEST_TEXT_SIZE (PASSED): Verified that I am able to rescale the text size from 9 to 72.
TEST_CLEAR_TEXT (PASSED): Able to clear using Clear Text option.
TEST_INFO (PASSED): Verified the Keyman Info option from Get Started menu and in the Home Screen.
TEST_SETTINGS (PASSED): Verified the Keyman Settings functionality and it is working as expected.
TEST_EUROLATIN (PASSED): Verified the EuroLatin Keyman and it showing the expected results.
TEST_US_BASIC (PASSED): Verified the English: US Basic as well as the Tagalog: US Basic and both are working as expected.

bharanidharanj commented 1 year ago

SUITE_MENU_FUNCTIONALITY: Menu Functionality Tests, in-app

GROUP_IPAD_LANDSCAPE: landscape mode in iPad

TEST_GET_STARTED (PASSED): Tested with Keyman 16.0.107-beta build in iOS 15.0 / iPad Air Simulator and the Get Started functionality is working as expected.
TEST_ADD_NEW_KEYBOARD (PASSED): Verified that we are able to install a new keyboard as per the instruction.
TEST_ADD_NEW_KEYBOARD_FROM_FILE (PASSED): Verified that we able to install a new keyboard from file.
TEST_SWITCH_KEYBOARD (PASSED): Able to switch between keyboards using the Keyboard picker menu.
TEST_SHARE (PASSED): Able to share the text with some other applications like Message App.
TEST_KEYMAN_BROWSER (PASSED): Verified that the predictive suggestions are properly applied when selected English - EuroLatin SIL keyboard.
TEST_TEXT_SIZE (PASSED): Verified that I am able to rescale the text size from 9 to 72.
TEST_CLEAR_TEXT (PASSED): Able to clear using Clear Text option.
TEST_INFO (PASSED): Verified the Keyman Info option from Get Started menu and in the Home Screen.
TEST_SETTINGS (PASSED): Verified the Keyman Settings functionality and it is working as expected.
TEST_EUROLATIN (PASSED): Verified the EuroLatin Keyman and it showing the expected results.
TEST_US_BASIC (PASSED): Verified the English: US Basic as well as the Tagalog: US Basic and both are working as expected.

bharanidharanj commented 1 year ago

SUITE_SYSTEM_KEYBOARD:

GROUP_IPHONE_PORTRAIT: portrait mode in iPhone

TEST_BASIC_USAGE (PASSED): Tested this as per the instructions with some of the keyman keyboards (eg., khmer_angkor, cameroon_qwerty, tamilzha 99 etc.,) and I was able to use it in all layers of the keyboard.
TEST_LONG_PRESS (PASSED): Verified that long press on all applicable layers are working as expected.
TEST_GLOBE_KEY (FAILED): Long Pressing a Globe key is not working in iPhone 12 Pro Simulator / iOS 15.0.

GROUP_IPHONE_LANDSCAPE: landscape mode in iPhone

TEST_BASIC_USAGE (PASSED): Tested this as per the instructions with some of the keyman keyboards (eg., khmer_angkor, cameroon_qwerty, tamilzha 99 etc.,) and I was able to use it in all layers of the keyboard.
TEST_LONG_PRESS (PASSED): Verified that long press on all applicable layers are working as expected.
TEST_GLOBE_KEY (FAILED): Long Pressing a Globe key is not working in iPhone 12 Pro Simulator / iOS 15.0.

bharanidharanj commented 1 year ago

SUITE_SWITCH_LOCALE:

GROUP_IPHONE_PORTRAIT: portrait mode in iPhone

TEST_KM_LOCALE (PASSED): Verified that the UI language has been changed to Khmer in Keyman App.
TEST_AM_LOCALE (PASSED): Verified that the UI language has been changed to Amharic in Keyman App.

GROUP_IPHONE_LANDSCAPE: landscape mode in iPhone

TEST_KM_LOCALE (PASSED): Verified that the UI language has been changed to Khmer in Keyman App.
TEST_AM_LOCALE (PASSED): Verified that the UI language has been changed to Amharic in Keyman App.

mcdurdin commented 1 year ago

@sgschantz this is a release-blocking issue, I think:

🟥 TEST_GLOBE_KEY (https://github.com/keymanapp/keyman/issues/7618#issuecomment-1323624997): Long Pressing a Globe key is not working in iPhone 12 Pro Simulator / iOS 15.0. (https://github.com/keymanapp/keyman/issues/7618#issuecomment-1323624997)

jahorton commented 1 year ago

@sgschantz this is a release-blocking issue, I think:

🟥 TEST_GLOBE_KEY (#7618 (comment)): Long Pressing a Globe key is not working in iPhone 12 Pro Simulator / iOS 15.0. (#7618 (comment))

I'm not seeing this on my iPhone SE, which is still on iOS 15.5. Identical behavior for both stable and beta (16.0.109) versions - the longpress menu displays as usual in the system keyboard.

sgschantz commented 1 year ago

The globe key works for me on real devices and on the simulator with beta and older versions. Based on the screenshot, I'm thinking that @bharanidharanj tested this in-app instead of with the system keyboard?

Here's what I saw in the simulator with the beta:

jahorton commented 1 year ago

The globe key works for me on real devices and on the simulator with beta and older versions. Based on the screenshot, I'm thinking that @bharanidharanj tested this in-app instead of with the system keyboard?

Right, the test did say "system keyboard", but those are clearly in-app screenshots. Longpresses don't do anything in-app until released, and even then, it just matches standard tap behavior.

Should probably ask for those two to be retested then.

sgschantz commented 1 year ago

@keymanapp-test-bot retest SUITE_SYSTEM_KEYBOARD GROUP_IPHONE_PORTRAIT TEST_GLOBE_KEY

sgschantz commented 1 year ago

@keymanapp-test-bot retest SUITE_SYSTEM_KEYBOARD GROUP_IPHONE_LANDSCAPE TEST_GLOBE_KEY

bharanidharanj commented 1 year ago

The globe key works for me on real devices and on the simulator with beta and older versions. Based on the screenshot, I'm thinking that @bharanidharanj tested this in-app instead of with the system keyboard?

Here's what I saw in the simulator with the beta:

Oops! Sorry! I will retest it and will post my result. :-)

bharanidharanj commented 1 year ago

SUITE_SYSTEM_KEYBOARD:

GROUP_IPHONE_PORTRAIT: portrait mode in iPhone

TEST_GLOBE_KEY (PASSED): Retested this as per Shawn's suggestion and now I am able to see the Keyboard picker menu while long pressing the key in a browser.

GROUP_IPHONE_LANDSCAPE: landscape mode in iPhone

TEST_GLOBE_KEY (PASSED): Retested this as per Shawn's suggestion and now I am able to see the Keyboard picker menu while long pressing the key in a browser.

bharanidharanj commented 1 year ago

SUITE_SYSTEM_KEYBOARD:

GROUP_IPAD_PORTRAIT: portrait mode in iPad

TEST_BASIC_USAGE (PASSED): Tested with Keyman 16.108-beta build in iPad Air (4th generation) / iOS 15.0 Simulator and I was able to use all the layers of the keyboards (khmer_angkor, sil_eurolatin and cameroon qwerty) without having any issues.
TEST_LONG_PRESS (PASSED): Verified that long press on all applicable layers are working as expected.
TEST_GLOBE_KEY (PASSED): After changing Keyman as the System wide keyboard, I was able to see the Keyboard picker menu while long pressing the key in a browser.

GROUP_IPAD_LANDSCAPE: landscape mode in iPad

TEST_BASIC_USAGE (PASSED): Verified in all layers of the keyboards (khmer_angkor, sil_eurolatin and cameroon qwerty) without having any issues.
TEST_LONG_PRESS (PASSED): Verified that long press on all applicable layers are working as expected.
TEST_GLOBE_KEY (PASSED): After changing Keyman as the System wide keyboard, I was able to see the Keyboard picker menu while long pressing the key in a browser.

bharanidharanj commented 1 year ago

SUITE_SWITCH_LOCALE:

GROUP_IPAD_PORTRAIT: portrait mode in iPad

TEST_KM_LOCALE (PASSED): Verified that the UI language has been changed to Khmer in Keyman App.
TEST_AM_LOCALE (PASSED): Verified that the UI language has been changed to Amharic in Keyman App.

GROUP_IPAD_LANDSCAPE: landscape mode in iPad

TEST_KM_LOCALE (PASSED): Verified that the UI language has been changed to Khmer in Keyman App.
TEST_AM_LOCALE (PASSED): Verified that the UI language has been changed to Amharic in Keyman App.

bharanidharanj commented 1 year ago

SUITE_INSPECT: Visual Inspection of master branch

TEST_VERIFY_UNIT_TESTS (PASSED): Tested with Keyman 16.0.108-beta build in all the areas in iPhone and iPad Simulators and all the tests are passed. Thanks.

jahorton commented 1 year ago

The one last thing I'd suggest - for @sgschantz: iOS does have a set of unit tests ready, but they're not attached to our CI.

You might want to give them a run and make sure they're still passing as expected.

keymanapp / keyman

test(iOS): Regression Test for Keyman 16 - Beta #7618

Keyman for iPhone and iPad Acceptance Test Procedures

User Testing

Gather Assets for Testing

Setup Steps

SUITE_INSPECT: Visual Inspection of master branch

SUITE_MENU_FUNCTIONALITY: Menu Functionality Tests, in-app

Test Cases

SUITE_SYSTEM_KEYBOARD:

SUITE_SWITCH_LOCALE

Test Cases

[TODO] SUITE_PREDICTIVE_TEXT: Predictive Text

User Test Results

✅ SUITE_INSPECT: Visual Inspection of master branch

✅ SUITE_MENU_FUNCTIONALITY: Menu Functionality Tests, in-app

✅ SUITE_SYSTEM_KEYBOARD:

✅ SUITE_SWITCH_LOCALE:

SUITE_INSPECT: Visual Inspection of master branch

SUITE_INSPECT: Visual Inspection of master branch

SUITE_MENU_FUNCTIONALITY: Menu Functionality Tests, in-app

GROUP_IPHONE_PORTRAIT: portrait mode in iPhone

SUITE_MENU_FUNCTIONALITY: Menu Functionality Tests, in-app

GROUP_IPHONE_PORTRAIT: portrait mode in iPhone

SUITE_MENU_FUNCTIONALITY: Menu Functionality Tests, in-app

GROUP_IPHONE_LANDSCAPE: landscape mode in iPhone

SUITE_MENU_FUNCTIONALITY: Menu Functionality Tests, in-app

GROUP_IPAD_PORTRAIT: portrait mode in iPad

SUITE_MENU_FUNCTIONALITY: Menu Functionality Tests, in-app

GROUP_IPAD_LANDSCAPE: landscape mode in iPad

SUITE_SYSTEM_KEYBOARD:

GROUP_IPHONE_PORTRAIT: portrait mode in iPhone

GROUP_IPHONE_LANDSCAPE: landscape mode in iPhone

SUITE_SWITCH_LOCALE:

GROUP_IPHONE_PORTRAIT: portrait mode in iPhone

GROUP_IPHONE_LANDSCAPE: landscape mode in iPhone

SUITE_SYSTEM_KEYBOARD:

GROUP_IPHONE_PORTRAIT: portrait mode in iPhone

GROUP_IPHONE_LANDSCAPE: landscape mode in iPhone

SUITE_SYSTEM_KEYBOARD:

GROUP_IPAD_PORTRAIT: portrait mode in iPad

GROUP_IPAD_LANDSCAPE: landscape mode in iPad

SUITE_SWITCH_LOCALE:

GROUP_IPAD_PORTRAIT: portrait mode in iPad

GROUP_IPAD_LANDSCAPE: landscape mode in iPad

SUITE_INSPECT: Visual Inspection of master branch