MakaraSok commented 1 year ago

The basic idea is to load the app (or test page), and then test input with latest release version of Keyman apps and a variety of keyboards.

Gather Assets for Testing

Install the following keyboards and have the test page link ready to load on the respective browser.

Keyman app - Latest version (please record the version in the test results)
Khmer Angkor keyboard package
EuroLatin (SIL) keyboard pagekage
Galaxie Hebrew (Positional) keyboard package
IPA (SIL) keyboard package
USX Editor test page: delete the first line and type in the text area to test

User Testing

The app may need to be tested in multiple environments. Each environment can be defined as a group here. For example, if you wanted to test a web app, you may have the following groups:

GROUP_IOS_SAFARI: iOS / Safari browser
GROUP_IOS_CHROME: iOS / Chrome browser
GROUP_ANDROID_CHROME: Android / Chrome browser
GROUP_ANDROID_FIREFOX: Android / Firefox

Load "the app or 'test page' being tested" on the respective OS (and browser) to test the followings:

TEST_KHMER_ANGKOR: Khmer Angkor
- Ensure longpress on ស and choose ឞ to produce ឞ.
- Type ស ុ ្រ (longpress of រ) ក to produce ស្រុក.
TEST_SIL_EURO_LATIN: SIL Euro Latin
- Longpress on a and choose ǎ to output ǎ
TEST_GALAXIE_HEBREW: Galaxie Hebrew mentioned above and here is how to type each key.
- Type b \ = r kbd>@</kbd a v ! y t to output בְּרֵאשִׁית. FAILED if the output is different.
TEST_SIL_IPA: sil_ipa
- Ensure the key sequence n > (long press of ] on 123 layer) produces ŋ.
TEST_INTERACTION_WITH_TEXT_FORMATTING:
- (n/a for the time being)
TEST_CJK_IMX:
- (n/a for the time being)

keymanapp-test-bot[bot] commented 1 year ago

User Test Results

Test specification and instructions

🟩 GROUP_IOS_SAFARI: iOS / Safari browser
- ✅ TEST_KHMER_ANGKOR (PASSED): Tested with the latest Keyman 17.0.69-alpha build in iOS 16.2 \ iPhone 14 Pro Max Simulator in Firefox browser and it is showing the correct output. (notes)
- ✅ TEST_SIL_EURO_LATIN (PASSED): Tested with the latest Keyman 17.0.69-alpha build in iOS 16.2 \ iPhone 14 Pro Max Simulator in Firefox browser and it is showing the correct output. (notes)
- ✅ TEST_GALAXIE_HEBREW (PASSED): Tested with the latest Keyman 17.0.69-alpha build in iOS 16.2 \ iPhone 14 Pro Max Simulator in Firefox browser and it is showing the correct output. (notes)
- ✅ TEST_SIL_IPA (PASSED): Tested with the latest Keyman 17.0.69-alpha build in iOS 16.2 \ iPhone 14 Pro Max Simulator in Firefox browser and it is showing the correct output. (notes)
- 🟩 TEST_INTERACTION_WITH_TEXT_FORMATTING (SKIPPED): Skipped this test since it is not ready for testing.
- 🟩 TEST_CJK_IMX (SKIPPED): Skipped this test since it is not ready for testing.
🟩 GROUP_IOS_CHROME: iOS / Chrome browser
- ✅ TEST_KHMER_ANGKOR (PASSED)
- ✅ TEST_SIL_EURO_LATIN (PASSED)
- ✅ TEST_GALAXIE_HEBREW (PASSED)
- ✅ TEST_SIL_IPA (PASSED)
- 🟩 TEST_INTERACTION_WITH_TEXT_FORMATTING (SKIPPED): Chrome browser is pretty similar to Safari browser. Hence, the results I have posted in Safari browser will be the same in Chrome If run it. Since, I don't have chrome browser in the iPhone Simulator, I skipped the test.
- 🟩 TEST_CJK_IMX (SKIPPED): Chrome browser is pretty similar to Safari browser. Hence, the results I have posted in Safari browser will be the same in Chrome If run it. Since, I don't have chrome browser in the iPhone Simulator, I skipped the test.
🟩 GROUP_ANDROID_CHROME: Android / Chrome browser
- ✅ TEST_KHMER_ANGKOR (PASSED)
- ✅ TEST_SIL_EURO_LATIN (PASSED)
- ✅ TEST_GALAXIE_HEBREW (PASSED)
- ✅ TEST_SIL_IPA (PASSED)
- 🟩 TEST_INTERACTION_WITH_TEXT_FORMATTING (SKIPPED): Skipped the test since it is not ready for testing.
- 🟩 TEST_CJK_IMX (SKIPPED): Skipped this test since it is not ready for testing.
🟥 GROUP_ANDROID_FIREFOX: Android / Firefox
- 🟥 TEST_KHMER_ANGKOR (FAILED): Tested with the latest Keyman 17.0.86-alpha build along with Firefox Focus browser in Android 12 / API 31 emulator and here is my observation: 1. After selecting the subkey menu from the Khmer Angkor keyboard, the cursor misplaced with some other letter. Seems to be an issue. (notes)
- ✅ TEST_SIL_EURO_LATIN (PASSED): Tested in the Firefox Focus browser in Android 12 / API 31 emulator and it is showing the correct output. (notes)
- ✅ TEST_GALAXIE_HEBREW (PASSED): Tested in the Firefox Focus browser in Android 12 / API 31 emulator and it is showing the correct output. (notes)
- 🟥 TEST_SIL_IPA (FAILED): After pressing > symbol from the keyboard, the whole letter disappear from the text area. Seems to be an issue. (notes)
- 🟩 TEST_INTERACTION_WITH_TEXT_FORMATTING (SKIPPED): Skipped this test since it is not ready for testing.
- 🟩 TEST_CJK_IMX (SKIPPED): Skipped this test since it is not ready for testing.

Retesting Template

``` @keymanapp-test-bot retest GROUP_ANDROID_FIREFOX TEST_KHMER_ANGKOR TEST_SIL_IPA ```

bharanidharanj commented 1 year ago

GROUP_ANDROID_CHROME: Android / Chrome browser

TEST_KHMER_ANGKOR (PASSED): Tested with the latest Keyman 17.0.69-alpha build in Android 12 / API 31 emulator in Chrome browser and it is showing the correct output.
TEST_SIL_EURO_LATIN (PASSED): Tested with the latest Keyman 17.0.69-alpha build in Android 12 / API 31 emulator in Chrome browser and it is showing the correct output.
TEST_GALAXIE_HEBREW (PASSED): Tested with the latest Keyman 17.0.69-alpha build in Android 12 / API 31 emulator in Chrome browser and it is showing the correct output.
TEST_SIL_IPA (FAILED): Tested with the latest Keyman 17.0.69-alpha build in Android 12 / API 31 emulator in Chrome browser and it is showing the wrong output.
TEST_INTERACTION_WITH_TEXT_FORMATTING (SKIPPED): Skipped the test since it is not ready for testing.
TEST_CJK_IMX (SKIPPED): Skipped this test since it is not ready for testing.

bharanidharanj commented 1 year ago

GROUP_ANDROID_FIREFOX: Android / Firefox

TEST_KHMER_ANGKOR (PASSED): Tested with the latest Keyman 17.0.69-alpha build in Android 12 / API 31 emulator in Firefox browser and it is showing the correct output.
TEST_SIL_EURO_LATIN (PASSED): Tested with the latest Keyman 17.0.69-alpha build in Android 12 / API 31 emulator in Firefox browser and it is showing the correct output.
TEST_GALAXIE_HEBREW (PASSED): Tested with the latest Keyman 17.0.69-alpha build in Android 12 / API 31 emulator in Firefox browser and it is showing the correct output.
TEST_SIL_IPA (FAILED): Tested with the latest Keyman 17.0.69-alpha build in Android 12 / API 31 emulator in Firefox browser and I noticed that letters are disappearing after I typed. Seems to be an issue.
TEST_INTERACTION_WITH_TEXT_FORMATTING (SKIPPED): Skipped this test since it is not ready for testing.
TEST_CJK_IMX (SKIPPED): Skipped this test since it is not ready for testing.

bharanidharanj commented 1 year ago

GROUP_IOS_SAFARI: iOS / Safari browser

TEST_KHMER_ANGKOR (PASSED): Tested with the latest Keyman 17.0.69-alpha build in iOS 16.2 \ iPhone 14 Pro Max Simulator in Firefox browser and it is showing the correct output.
TEST_SIL_EURO_LATIN (PASSED): Tested with the latest Keyman 17.0.69-alpha build in iOS 16.2 \ iPhone 14 Pro Max Simulator in Firefox browser and it is showing the correct output.
TEST_GALAXIE_HEBREW (PASSED): Tested with the latest Keyman 17.0.69-alpha build in iOS 16.2 \ iPhone 14 Pro Max Simulator in Firefox browser and it is showing the correct output.
TEST_SIL_IPA (PASSED): Tested with the latest Keyman 17.0.69-alpha build in iOS 16.2 \ iPhone 14 Pro Max Simulator in Firefox browser and it is showing the correct output.
TEST_INTERACTION_WITH_TEXT_FORMATTING (SKIPPED): Skipped this test since it is not ready for testing.
TEST_CJK_IMX (SKIPPED): Skipped this test since it is not ready for testing.

bharanidharanj commented 1 year ago

Test Results

GROUP_IOS_CHROME: iOS / Chrome browser

TEST_KHMER_ANGKOR (SKIPPED): Chrome browser is pretty similar to Safari browser. Hence, the results I have posted in Safari browser will be the same in Chrome If run it. Since, I don't have chrome browser in the iPhone Simulator, I skipped the test.
TEST_SIL_EURO_LATIN (SKIPPED): Chrome browser is pretty similar to Safari browser. Hence, the results I have posted in Safari browser will be the same in Chrome If run it. Since, I don't have chrome browser in the iPhone Simulator, I skipped the test.
TEST_GALAXIE_HEBREW (SKIPPED): Chrome browser is pretty similar to Safari browser. Hence, the results I have posted in Safari browser will be the same in Chrome If run it. Since, I don't have chrome browser in the iPhone Simulator, I skipped the test.
TEST_SIL_IPA (SKIPPED): Chrome browser is pretty similar to Safari browser. Hence, the results I have posted in Safari browser will be the same in Chrome If run it. Since, I don't have chrome browser in the iPhone Simulator, I skipped the test.
TEST_INTERACTION_WITH_TEXT_FORMATTING (SKIPPED): Chrome browser is pretty similar to Safari browser. Hence, the results I have posted in Safari browser will be the same in Chrome If run it. Since, I don't have chrome browser in the iPhone Simulator, I skipped the test.
TEST_CJK_IMX (SKIPPED): Chrome browser is pretty similar to Safari browser. Hence, the results I have posted in Safari browser will be the same in Chrome If run it. Since, I don't have chrome browser in the iPhone Simulator, I skipped the test.

MakaraSok commented 1 year ago

GROUP_IOS_CHROME: iOS / Chrome browser

Tested with the latest Keyman 17.0.86-alpha build in iOS 16.4.1 / iPad mini 6 and it is showing the correct output. Chrome version 112.0.5615.46

TEST_KHMER_ANGKOR (PASSED):
TEST_SIL_EURO_LATIN (PASSED):
TEST_GALAXIE_HEBREW (PASSED):
TEST_SIL_IPA (PASSED):

GROUP_ANDROID_CHROME: Android / Chrome browser

Tested with the latest Keyman 17.0.86-alpha build in Android 10 / Samsung Galaxy S9+ and it is showing the correct output. Chrome version 112.0.5615.48

TEST_KHMER_ANGKOR (PASSED):
TEST_SIL_EURO_LATIN (PASSED):
TEST_GALAXIE_HEBREW (PASSED):
TEST_SIL_IPA (PASSED):

MakaraSok commented 1 year ago

@keymanapp-test-bot retest GROUP_ANDROID_FIREFOX TEST_KHMER_ANGKOR TEST_SIL_EURO_LATIN TEST_GALAXIE_HEBREW TEST_SIL_IPA

MakaraSok commented 1 year ago

@bharanidharanj Could you please help and retest this group as it was not behaving as expected for me on Android 10.

bharanidharanj commented 1 year ago

@bharanidharanj Could you please help and retest this group as it was not behaving as expected for me on Android 10.

@MakaraSok Sure. I will test it.

bharanidharanj commented 1 year ago

Test Results

GROUP_ANDROID_FIREFOX: Android / Firefox

TEST_KHMER_ANGKOR (FAILED): Tested with the latest Keyman 17.0.86-alpha build along with Firefox Focus browser in Android 12 / API 31 emulator and here is my observation: 1. After selecting the subkey menu from the Khmer Angkor keyboard, the cursor misplaced with some other letter. Seems to be an issue.
TEST_SIL_EURO_LATIN (PASSED): Tested in the Firefox Focus browser in Android 12 / API 31 emulator and it is showing the correct output.
TEST_GALAXIE_HEBREW (PASSED): Tested in the Firefox Focus browser in Android 12 / API 31 emulator and it is showing the correct output.
TEST_SIL_IPA (FAILED): After pressing > symbol from the keyboard, the whole letter disappear from the text area. Seems to be an issue.