Create tests for APG design pattern: checkbox

[zcorpan]

APG design pattern: https://w3c.github.io/aria-practices/#checkbox

• [x] Test committed: https://w3c.github.io/aria-at/review/checkbox.html
• Test reviewed:
  • [x] Review of test description
  • [x] JAWS commands
  • [x] NVDA commands
  • [x] VoiceOver commands
• Test results
  • [x] JAWS/Chrome
  • [ ] JAWS/Firefox
  • [ ] NVDA/Chrome
  • [x] NVDA/Firefox
  • [ ] Mac VoiceOver/Chrome
  • [x] Mac VoiceOver/Safari
• [ ] TODO: discuss assertions related to the group role and name.

[mfairchild365]

I started my review and am making comments on Google Drive. I'll make a PR with changes. Notice the section titled 'questions to ask during the review'. Might be good to add a similar section to our review output to help facilitate consistent reviews?

[mfairchild365]

@mcking65 - I noticed that there are some commands that require Smart Nav is on in JAWS. As written, I think this may confuse testers, since Smart Nav is off in JAWS by default (but can be toggled via inert+x) and we do not currently describe how to turn it on or off.

A few options:

1. Describe how to toggle Smart Nav mode
2. Write separate tests for Smart Nav since enabling it requires a precondition
3. Do not test Smart Nav right now (this would be my preference since it reduces complexity and the setting is turned off my default)

Thoughts?

[mfairchild365]

Another question: There are assertions for The role 'group' is conveyed. I think this is worded well, however, most screen readers imply the group role when jumping to a form control within the group by only announcing the name of the group, followed by the rest of the information about the control. The presence of that additional name also conveys the group role.

I think there is an opportunity for us to describe different ways that screen readers might imply the role and thus successfully convey the role. As written, I think many testers might not know that behavior is okay and meets the requirement of 'conveyed'.

Do you agree? Have any thoughts about how we could address this?

[mfairchild365]

TODO

• [x] simplify the change in state assertions and improve associated instructions.
• [x] remove the word 'the' that prefixes all assertions

[mcking65]

Review of Checkbox Tests

This is actually looking very good. I think we have the right tests and the right assertions. Nearly all of this feedback is editorial. I think we need to discuss the wording of only one assertion related to checkbox grouping.

Editorial issues in multiple tests

Issue 1: The instructions say:

Then, perform the task "navigate through boundaries of checkbox group" using the following commands:

Since this is a list and since the task already has active wording, we do not need the phrase "Then, perform the task ". And, we do not need quote marks around the task text. It would be easier to read the plan without the extra words and punctuation.

That is, it could be written as:

Navigate through boundaries of checkbox group using the following commands:

It would be good if we capitalize the first word of the tasks in the WPT file.

Issue 2: The word "instructions" is misspelled as "instuctions" under scripted instructions. There are 17 occurences on the checkbox test review page.

Issue 3: This may be something to address later. The only time that VoiceOver quick nav needs to be off for interaction mode to work is if either the user needs to type in an edit field or if the arrow keys are in the list of interaction mode commands that need to be tested. If the user does not know to turn them off for an edit field, the user is not qualified to run tests. However, if the command list includes arrow keys, it would be good to include the instruction, "1. Toggle Quick Nav OFF by pressing the Left Arrow and Right Arrow keys at the same time." Otherwise, we really don't need that instruction, and it seems kind of discrediting to include it. For now, it's not really a big deal.

Assertion about group boundaries in Test 1

Test 1: Navigate through a checkbox group in interaction mode.

Boundaries of the group (before the first checkbox and after the last checkbox) are conveyed

We might want to consider more flexible wording for this. Perhaps:

Users can discern whether a checkbox is a member of the group.

Thus, the screen reader does not necessarily have to convey the boundary directly. For instance, if it announced the group name for each checkbox in the group, the assertion would be satisfied. NVDA tells users when leaving entities, but JAWS does not. This would be fine if the user could readily find out if a checkbox is a member of the group, and they may choose to do that in some way other than having an announcement that you are leaving the group. Personally, NVDA's "leavaing" announcements are something I find more problematic than helpful. They get in the way of hearing the newly focused element.

I'm not sure what the best answer here is. This might actually be one to discuss with developers of screen readers that do not choose to announce the end of elements when tabbing.

Test 1 VoiceOver commands

I don't think we need to test:

• Control+Option+A: this reads continuously and is not in any way specific to any type of control. It is like insert+down in JAWS or NVDA. BTW, the VoiceOver help for this command is rather misleading.
• Control+U to open the ‘Rotor’ menu; then press the Right arrow until the list of 'Form Controls’ appears; then select a checkbox in the list of form controls using the Up/Down arrow keys; then press ‘Enter’ to navigate to that checkbox

The rotor test is equivalent to ctrl-cmd-opt-j. BTW, this is also true for JAWS and NVDA. We do not need to test the element list commands, e.g., insert+F5 with JAWS or insert+f7 with NVDA.

Applies to and Commands for Test 2

Test 2: Navigate through a checkbox group in reading mode.

• Set "Applies to" to "JAWS, NVDA"
• For JAWS, Add: X and Shift+X
• For JAWS, remove: Control+Insert+X to see a list of checkboxes; then use the Up/Down arrows to select a checkbox; then press Enter to navigate to that checkbox
• For NVDA, Remove:
  • Insert+Down (or CapsLock+Down)
  • Insert+F7 (or CapsLock+F7) to access list of elements on the page; then select ‘type’ = ‘form fields’; then select a checkbox in the list of form controls; then press ‘Enter’ to navigate to that checkbox

Test 3: Navigate to a checked checkbox in interaction mode.

For VoiceOver, remove commands:

• Control+Option+A
• Control+U to open the ‘Rotor’ menu; then press the Right arrow until the list of 'Form Controls’ appears; then select a checkbox in the list of form controls using the Up/Down arrow keys; then press ‘Enter’ to navigate to that checkbox

Test 4: Navigate to a checked checkbox in reading mode.

• Change "Applies to" to "JAWS, NVDA"
• In JAWS commands, remove: Control+Insert+X to see a list of checkboxes; then use the Up/Down arrows to select a checkbox; then press Enter to navigate to that checkbox
• In NVDA commands, remove:
  • Insert+Down (or CapsLock+Down)
  • Insert+F7 (or CapsLock+F7) to access list of elements on the page; then select ‘type’ = ‘form fields’; then select a checkbox in the list of form controls; then press ‘Enter’ to navigate to that checkbox

Test 5: Navigate to an unchecked checkbox in interaction mode.

• In VoiceOver commands, remove:
  • Control+Option+A
  • Control+U to open the ‘Rotor’ menu; then press the Right arrow until the list of 'Form Controls’ appears; then select a checkbox in the list of form controls using the Up/Down arrow keys; then press ‘Enter’ to navigate to that checkbox

Test 6: Navigate to an unchecked checkbox in reading mode.

• Change "Applies to" to "JAWS, NVDA"
• In JAWS commands, remove: Control+Insert+X to see a list of checkboxes; then use the Up/Down arrows to select a checkbox; then press Enter to navigate to that checkbox
• In NVDA commands, remove:
  • Insert+Down (or CapsLock+Down)
  • Insert+F7 (or CapsLock+F7) to access list of elements on the page; then select ‘type’ = ‘form fields’; then select a checkbox in the list of form controls; then press ‘Enter’ to navigate to that checkbox

Test 8: Operate a checkbox in reading mode.

• Change "Applies to" to "JAWS, NVDA"
• NVDA instructions say: "1. Put NVDA into Browser Mode using Escape". The mode is called "Browse" not "Browser". And, the more reliable command would be "Insert+Space".

* Test 10: Read a checkbox group in reading mode.

• Change "Applies to" to "JAWS, NVDA"
• Same comment as above about instruction "1. Put NVDA into Browser Mode using Escape"

Test 11: Read an unchecked checkbox in reading mode.

• the title is incorrect: It says the mode is reading but it is actually interaction.

Test 12: Read an unchecked checkbox in interaction mode.

• The title is incorrect: It says the mode is interaction, but it is actually reading.
• Change "applies to" to "JAWS, NVDA"
• Same comment as above about NVDA instruction: "1. Put NVDA into Browser Mode using Escape"

Test 14: Read a checked checkbox in reading mode.

• Change "Applies to" to "JAWS, NVDA"
• Same comment as above about NVDA instruction: "1. Put NVDA into Browser Mode using Escape"

Test sequencing

It would be easier to follow the plan if the tests were in the following order:

• Test 1: Navigate through a checkbox group in interaction mode.
• Test 2: Navigate through a checkbox group in reading mode.
• Test 9: Read a checkbox group in interaction mode.
• Test 10: Read a checkbox group in reading mode.
• Test 5: Navigate to an unchecked checkbox in interaction mode.
• Test 6: Navigate to an unchecked checkbox in reading mode.
• Test 12: Read an unchecked checkbox in interaction mode.
• Test 11: Read an unchecked checkbox in reading mode.
• Test 3: Navigate to a checked checkbox in interaction mode.
• Test 4: Navigate to a checked checkbox in reading mode.
• Test 13: Read a checked checkbox in interaction mode.
• Test 14: Read a checked checkbox in reading mode.
• Test 7: Operate a checkbox in interaction mode.
• Test 8: Operate a checkbox in reading mode.

[mfairchild365]

Thank you, @mcking65

• [x] 1. re issue 1: I agree.
• [x] 2. re issue 2: I agree.
• [ ] 3. re issue 3: From my experience, the more explicit and the less abstract test instructions are, there will be less room for interpretation and more consistency in results. So, I think having instructions about turning QuickNav off is good.
• [x] 4. re language of boundaries assertion - This is an interesting idea. I'll think on it some more. - leave for now and discuss later.
• [x] 5. re "Browse" mode typo - I agree.
• [x] 6. I agree that we don't need to test commands to read all content.
• [x] 7. I hesitate to remove the commands for element lists. I have found situations where elements are not listed, while they can be found with the command to jump to the next element of type (ctrl-cmd-opt-j.). It's fairly rare, but it does happen. This really speaks to where we draw the line with how many and which commands we test. Personally, I think the fewer the better, as long as the commands are fairly representative. - remove for now and discuss
• [x] 8. Re incorrect titles - thanks! I'll get that fixed.
• [x] 9. Re changes to 'applies to': I agree. Good catch.
• [ ] 10. Test sequencing - I agree, but I am unsure of how to change the ordering.
• [x] 11. Yohta's feedback: test F3 and F4 (worked)

[ghost]

Aside from conversations already made between Matt and Michael, here are a few thoughts and findings from my review.

Review from test settings

• I wasn't quite sure the idea of DOM is clear for SR users. Is it something we can assume most SR users are aware of?

Review from individual tests, VO in particular

• No key commands are available in Test 4,6,8,10,12,14

• "Option" key is missing to open the rotor Test 1,3,5

• commands (F3 and F4) didn't work, in my environment, though I may not be understanding the Scripted Instructions "Put DOM focus on the Scripted Instructions first checkbox" right. (Test 9,11)

[mfairchild365]

@Yohta89 - for f3 and f4 you may also need to press the 'fn' key depending on what keyboard you are using and configuration options for your keyboard.

[ghost]

Tested VoiceOver with Safari, and came across two quick clarifications before moving on to Chrome!

1. Does the test result instantly reflected in the review page, or is it stored in a different place? I only see two results of JAWS , and not seeing results from VoiceOver in the page I’m referring.*Attached json file in email

2. Upon submitting the tests, screen reader output information in the “Detail” column in the individual test result table was rendered into JAWS, even though tested screen reader was VoiceOver. I’m not sure if this relates to first clarification. *Please see the attached .xlsx file for screenshot.

Result_page.xlsx

[mfairchild365]

Good questions. Test results are not automatically reflected. The Json file has to be reviewed and added to the project for that to happen. Issue 2 sounds like a bug.

Are we considering Chrome in scope for VoiceOver right now?

[ghost]

Test result from VoiceOver/Safari is in my google folder. Retrieve the VO/Safari file from here

There were two tasks (three sets of key commands) I marked as Fail. Test (1 of 14): Navigate through a checkbox group in interaction mode. Command: Control+Option+Command+J / Shift+Control+Option+Command+J Failing Assertions:

• Role 'group' is conveyed
• Group name 'Sandwich Condiments' is conveyed
• Boundaries of the group (before the first checkbox and after the last checkbox) are conveyed

Detail: VO speech output was “Lettuce, unchecked”, and I heard no information about group role, such as group name, boundary.

Test (9 of 14): Read a checkbox group in interaction mode. Command: Control+Option+F3 and Control+Option+F4 Failing Assertions:

• Role 'group' is conveyed
• Group's name 'Sandwich Condiments' is conveyed

Detail VO speech output were:

• “Lettuce, unchecked, checkbox, is in the VoiceOver cursor”, after key commands Control+Option+F3 and I heard no information about group name and role,

• “Lettuce, unchecked, checkbox, has keyboard focus”, after key commands Control+Option+F4 and I heard no information about group name and role,

[mcking65]

Unfortunately, I ran into a show-stopper with testing checkbox. I was wrong earlier; we are missing tests. The problems are that:

1. When a test asserts container-related behaviors, e.g., entering/leaving a container, we cannot combine tab and shift+tab; they need to be two separate commands for the task. The same is true for x and shift+x, f and shift+f, etc. This is only when testing the container. For example, Tabbing into the checkbox group from above gives different output from shift+tabbing into the group from below.
2. When we are testing container-related behaviors, navigating into and out of the container need to be separate tests. For example, JAWS announces group labels when navigating into the container, but it does not when navigating out. There is no way to capture that in the current test plan.

So, with the current plan, we cannot produce accurate results. We need some additional tests and slightly different command lists.

Here are my recommendations.

Change Test 1 and add an interaction mode test for leaving the group

Test 1: Navigate through a checkbox group in interaction mode.

1. Change this to: Test 1: Navigate into a checkbox group in interaction mode.
2. Change the command list so there are 2 commands that are tested separately: Tab, shift+Tab
3. Add Test: Navigate out of a checkbox group in interaction mode. This test has the same commands and assertions as the current test 1. However, make the assertions all "Nice to Have" priority.

Change Test 2 and add a reading mode test for leaving the group

Test 2: Navigate through a checkbox group in reading mode.

1. Change to: Test 2: Navigate into a checkbox group in reading mode.
2. Change the command list so that it has 6 commands that are tested separately: X, Shift+X, Up Arrow, Down Arrow, Tab, Shift+Tab
3. Add Test: Navigate out of a checkbox group in reading mode. This test has the same commands and assertions as the current test 2. However, make the assertions all "Nice to Have" priority.

Other changes

I think we should change the wording of these two test names:

• Test 9: Read a checkbox group in interaction mode.
• Test 10: Read a checkbox group in reading mode.

The test name implies you are going to read the complete group. I think this will be confusing, especially in reports. I recommend:

• Test 9: Read grouping information of a grouped checkbox in interaction mode.
• Test 10: Read grouping information of a grouped checkbox in reading mode.

Additionally, wording of the command should be changed. The current wording is:

When focus or cursor is on a checkbox, read the group

This should be changed to:

When focus or cursor is on a checkbox, read its grouping information

Finally, the checkbox tests do not have links to the APG pattern or to the page with the widget being tested. I think we should have links to both. I'll add this to #37.

[mcking65]

@Yohta89 wrote:

Test result from VoiceOver/Safari is in my google folder.

Thank you!

There were two tasks (three sets of key commands) I marked as Fail. Test (1 of 14): Navigate through a checkbox group in interaction mode. Command: Control+Option+Command+J / Shift+Control+Option+Command+J Failing Assertions:

• Role 'group' is conveyed
• Group name 'Sandwich Condiments' is conveyed
• Boundaries of the group (before the first checkbox and after the last checkbox) are conveyed

Detail: VO speech output was “Lettuce, unchecked”, and I heard no information about group role, such as group name, boundary.

You are correct, those assertions fail.

Tab and shift+tab also fail when leaving the group but not when entering. We don't have a way to capture that. See my previous comment.

I imagine that both Apple and Freedom Scientific have consciously decided not to announce leaving. So, this will be a point of discussion. In that case, they may not even agree that it would be nice to have. They might agree that it would be nice or appropriate if there was another checkbox that was outside the boundary of the group. Should be interesting discussion.

Test (9 of 14): Read a checkbox group in interaction mode. Command: Control+Option+F3 and Control+Option+F4 Failing Assertions:

• Role 'group' is conveyed
• Group's name 'Sandwich Condiments' is conveyed

Detail VO speech output were:

• “Lettuce, unchecked, checkbox, is in the VoiceOver cursor”, after key commands Control+Option+F3 and I heard no information about group name and role,
• “Lettuce, unchecked, checkbox, has keyboard focus”, after key commands Control+Option+F4 and I heard no information about group name and role,

Yes, correct, they fail.

[mcking65]

@mfairchild365 asks:

Are we considering Chrome in scope for VoiceOver right now?

For our meetings next week, we don't need Vo/Chrome. It is in scope for the project.

[mfairchild365]

@mcking65 - I created a PR to address the issues you listed.

[spectranaut]

ok Matt, thanks to @mfairchild365's speedy PR here are the changes reflected in the test plan: https://w3c.github.io/aria-at/review/checkbox.html

@Yohta89, any chance you can re-run just the checkbox grouping related tests and upload them?

[ghost]

Retested VO/Safari, and the result .json file is in my google doc again. @spectranaut I re-ran all the tests since I already had the previous test file. @mcking65 Can you remind me of the deadline to have results of JAWS/Chrome and NVDA/Firefox? Would it still be usable for your next meeting with SR developers if I finished JAWS and NVDA in this weekend?

Quick notes from VO/Safari tests: Task: Navigate into a checkbox group in interaction mode. Command: Control+Option+Command+J / Shift+Control+Option+Command+J Speech:Lettuce, unchecked
Note: VO didn't announce the group role, name, boundary upon pressing above key commands.

---

Task: Navigate out of a checkbox group in interaction mode. Command: Tab / Shift+Tab Speech:Keyboard Support/Checkbox Design Pattern, Link Note:The focus moved to a next/previous link without announcing the fact that the focus got out of the checkbox group, thus VO only announced next/previous link name without announcing the group role, name, boundary.

Task: Navigate out of a checkbox group in interaction mode. Command: Control+Option+Command+J / Shift+Control+Option+Command+J Speech:Last form elements, Sprouts
Note:I marked this test as Fail (No Output) . I didn't mark these assertion as 'Incorrect Output' , since my assumption for incorrect is misleading information. However, I'm also concerned about the validity of this command, since the focus didn't go out of the group and stuck in the first/last checkbox, so that there seemed to be no way VO announces group role, name, boundary.

---

Task: Read grouping information of a checkbox group in interaction mode. I think two assertions in this task still fail. My assumption is grouping information is its group's name 'Sandwich Condiments', and I didn't hear this information.

[mcking65]

ok Matt, thanks to @mfairchild365's speedy PR here are the changes reflected in the test plan: https://w3c.github.io/aria-at/review/checkbox.html

Looks awesome! Thank you, thank you @spectranaut and @mfairchild365!!

[mcking65]

@Yohta89 commented:

@mcking65 Can you remind me of the deadline to have results of JAWS/Chrome and NVDA/Firefox? Would it still be usable for your next meeting with SR developers if I finished JAWS and NVDA in this weekend?

We don't have the NVDA meeting booked, so we really only need JAWS/Chrome by Monday. It'd be good to have NVDA by Wednesday.

Your interpretations of the VO behavior are all correct.

[spectranaut]

Yesterday and today I wrote a script that can diff two results files and save a new one. @mcking65 , please try running it once you have finished your test run. To run:

1. run npm install in the aria-at directory after pulling master
2. go to the scripts directory
3. run node test-comparer.js path-to-file-1.json path-to-file-2.json --html

This will output the differences to the terminal and to a html file called "differences.html". The html file is pretty janky, I didn't have much time to think about formatting, sorry!

If you want to go through and save one result or the other, you can run with "--save". In this case, the program will stop after every difference and ask you which one you would like to save. And then if will output it to a new file with the name "revised-VoiceOver-X-Safari-Y.json" (where X and Y are the version numbers). If this script is not screen reader friendly for resolving difference, then could you, Matt, review the results via the html file and tell me which to save I can run it on Monday morning? I'll get it in before the meeting.

I don't have much time in my working day left but I'm going to address Matt's issue with the results page, next!

[ghost]

Tested JAWS/Chrome, and the result .json file is in my google doc.

There were several tests I marked them as fail, and following are detail. Note that last paragraph of Test(13,14,15,16) were full support, but just wanted to leave notes.

---

Test:(1): Navigate into a checkbox group in interaction mode. Command: Tab/Shift+Tab Speech: Sandwich Condiments. Lettuce checkbox. Not checked. Note: In both key commands, JAWS didn't announce Role 'group' and its boundaries.

---

Test(2): Navigate into a checkbox group in reading mode. Command: Up Arrow/Tab/Shift+Tab Note: JAWS didn't announce Role 'group' in commands above. Although JAWS did announce end boundary of group when I pressed "Up Arrow", it still didn't announce group name 'Sandwich Condiments'.

---

Test(3): Navigate out of a checkbox group in interaction mode. Command: Tab/Shift+Tab Note: JAWS didn't announce Role 'group', group name, and boundaries in commands above. When I pressed Tab/Shift+Tab, it just moved focus to the next focusable link without announcing any group information.

---

Test(4): Navigate out of a checkbox group in reading mode. Command: Down Arrow/Tab/Shift+TabU Note: JAWS didn't announce group name 'Sandwich Condiments' in commands above. Although JAWS did announce endof boundary of group when I pressed "DownArrow", it still didn't announce group name 'Sandwich Condiments'. When I pressed Tab/Shift+Tab,it just moved focus to the next focusable link without announcing any group information.

---

Test(11,12):Read grouping information of a checkbox group in interaction/reading mode. Command: Insert+Tab (or CapsLock+Tab)/Insert+Up (or CapsLock+I) Speech: Lettuce checkbox, not checked Note: JAWS didn't announce any role 'group and group name 'Sandwich Condiments' information.

---

Test(13,14,15,16): Read a (un)checked checkbox in interaction/reading mode. Note: I marked these tests as Full support, since they did work in primary key command Insert+Tab/Insert+Up. But wanted to flag that alternative command in parenthesis (or CapsLock+Tab/or CapsLock+I) didn't work on my Windows Desktop environment.

[ghost]

Tested NVDA/Firefox, and the result .json file is in my google doc.

The rationale of assertions I marked as Fail is the following:

---

test:Navigate into a checkbox group in reading mode. Command: Up Arrow/Down Arrow Speech:Lists with four items checkbox not checked Sprouts/Lists with four items checkbox not checked Lettuce Notes: No group role, name, and boundaries information was conveyed.

---

Test:Navigate out of a checkbox group in interaction mode. Command: Tab/Shift+Tab Speech:Keyboard Support button, collapsed/Checkbox Design Pattern, Link Notes:The focus moved to a next/previous link without announcing the fact that the focus got out of the checkbox group, thus NVDA only announced the next/previous link name without announcing the group role, name, boundary.

---

Test: Navigate out of a checkbox group in reading mode. Command: X/Shift+X/Up Arrow/Down Arrow/Tab/Shift+Tab Speech: No next checkbox/No previous checkbox/Out of the list, heading level three, Sandwich Condiments/Out of the list, button collapsed, Keyboard Support/Keyboard Support button, collapsed/Checkbox Design Pattern, Link Notes:None of the key commands resulted in announcing the group role, name, boundary.

---

Test:Read grouping information of a checkbox group in interaction mode. Command: Insert+Tab (or CapsLock+Tab)/Insert+Up (or CapsLock+Up) Speech:Lettuce checkbox focused, not checked/Lettuce Notes:No group role, name announced.

---

Test:Read grouping information of a checkbox group in reading mode. Command: Insert+Tab (or CapsLock+Tab)/Insert+Up (or CapsLock+Up) Speech:Lettuce checkbox focused, not checked/Checkbox not checked, Lettuce Notes:No group role, name was conveyed.

---

Test:Read an unchecked checkbox in interaction mode. Command: Insert+Up (or CapsLock+Up) Speech:Lettuce Notes:No checkbox role and state of the checkbox was conveyed.

---

Test:Read a checked checkbox in interaction mode. Command: Insert+Up (or CapsLock+Up) Speech:Lettuce Notes:No checkbox role and state of the checkbox was conveyed.

[mfairchild365]

@mcking65 - Should our test plan also include tests and assertions related to the list semantics that are present in the example?

[mcking65]

I did more changing in my PR #170 than I anticipated. The diff is big enough to be about useless so I wrote a summary of changes that you can use when reading the test plan. Please have a close look at the summary.

I included an experimental idea, a kind of shortcut, to solving the mode problem raised by Apple -- include versions of the test that do not mention mode and apply only to VoiceOver. I am curious to see how this is handled in the runner and harness. I am wondering if the test count/numbering is going to be based on the numbers in the file names? If so, that would be confusing when not every test applies to the AT the tester is currently running.

I can't figure out how to run the harness locally. localhost:3000 comes up empty after npm run start.

I got some errors from the create-tests script that I couldn't figure out how to resolve.

I am curious why we need the test IDs in the commands csv file. Seems like we could make things a lot simpler by using the command name, mode, and AT as the keys. In particular, reading a checkbox is theexact same command regardless of the state of the checkbox; same is true for navigating to the checkbox. We could have one command "Read Checkbox" instead of "Read Checked Checkbox" and "Read Unchecked Checkbox".

Re-reading this thread, I noticed that we haven't yet made a call on JAWS smart nav. I think I would be fine with removing that. We could include that change in this PR (assuming the PR isn't a disaster).

[robfentress]

Has there been any discussion of including tests for whether information is conveyed correctly when accessed using the Web Rotor (Control+Option+U) on VoiceOver or through the List of Form Elements (Insert+F5) on JAWS or can it safely be assumed that things will be communicated appropriately in these contexts? Similarly, should we be checking for whether things are communicated appropriately when navigating via other key commands Control+Option+Command+J or X? I understand if we've already got enough on our plate, but I thought I'd mention it in case nobody else has as something to be considered.

[robfentress]

Grrr. Nevermind. I see the Control+Option+Command+J and X. Sorry for not looking more carefully.