Initial Mode Switching Exploration

[jscholes]

Initial Mode Switching Exploration

Background

Within ARIA-AT tests targeting Windows screen readers that have multiple modes of operation (currently JAWS and NVDA), we include separate tests to evaluate behaviour in those separate modes where applicable. We have two abstract terms for these modes, "reading" and "interaction", illustrated via the following two example tests:

• Navigate forwards to a button in reading mode: the aim of this test is to move the JAWS Virtual PC Cursor and NVDA Browse Mode cursor to a button, in a forwards direction, using commands like B, F, and Down Arrow. The test also includes Tab as a command, to move system focus while the relevant screen reader cursor is active.
• Navigate forwards to a button in interaction mode: the aim of this test is to move only the system focus to the same button, likewise in a forwards direction, without the relevant reading cursor being active in the screen reader. The JAWS Virtual PC Cursor is switched off, NVDA is set to Focus Mode, and the only command used in the test is Tab. This way, we can spot any differences in behaviour when focus is moved in the two different modes.

With default settings, both JAWS and NVDA include functionality to automatically switch between reading and interaction modes when it is felt to be appropriate, or stay in a given mode when it isn't. For example:

• When moving the JAWS Virtual PC Cursor into a text (or "edit") field, the cursor is automatically disabled, allowing the user to type rather than printable characters being intercepted as quick navigation commands (like H for next heading).
• When the user navigates out of that same field, e.g. with the arrow keys or by pressing Tab, JAWS will either:
  • reenable the Virtual PC Cursor, e.g. when the user has arrowed to some static text or moved focus to a control that doesn't require interaction mode like a link; or
  • stay in interaction mode, and not reenable the Virtual PC Cursor just yet, e.g. because the user tabbed from one edit field to another.

Most ARIA-AT tests do not currently include any assertions aimed at evaluating mode switching or retention. To my knowledge, the only test plan that does is Editor Menubar Example, which includes a single test called:

Navigate to menubar with commands that switch mode from reading to interaction

And a single relevant assertion within that test, worded as:

Change of mode from reading to interaction is conveyed

Note that this test plan has not been modified since July 29th, 2020, almost three years ago at the time of writing. As such, it does not follow many of the test writing practices established since then, such as separated tests based on directionality, test title/assertion wording, heavy reliance on setup scripts, etc.

Objective and Rationale

Mode switching and retention, herein referred to as "mode selection", are important aspects of a screen reader user's experience:

1. Mode selection that is too frequent, not frequent enough, or otherwise not appropriate to the situation at hand can cause significant confusion and/or irritation.
2. When a particular control requires keyboard input that would usually be intercepted for purposes of reading mode commands, the lack of automatic mode switching may result in users becoming unclear about how to operate that control.
3. Point #2 also applies in reverse. After interacting with certain controls and then expecting to be returned to a reading context, retention of interaction mode may be disconcerting or represent a blocking issue if users do not know how to manually switch modes.

The aim, therefore, is to begin evaluating mode selection behaviour within our tests. This will represent a large scope of work, and hence our primary short-term goal is to determine:

• the short-and-longer-term scope of mode switching presence in our tests, i.e. what we do and don't need to do now and in the future;
• any changes to the test format that are felt to be helpful or required, along with relevant changes in the app or other tooling to support those changes; and
• any other implications not listed here.

Working Example

To aid in extrapolation of requirements, and discussion of same, this section presents a working example based on the test plan that targets the Editable Combobox with Both List and Inline Autocomplete example in the APG. The relevant test plan preview can be found at:

https://aria-at.netlify.app/review/combobox-autocomplete-both-updated.html

Note that this is not intended to be an exhaustive evaluation of how mode selection might be incorporated into an existing test plan, but rather some proposed starting points that can later be added to, subtracted from, and/or changed as needed. In particular, adaptation other test plans may end up being less or more complex than what is presented here.

Navigate to a combobox in reading mode

The title of this section represents a repeated testing task, abbreviated to remove variation between tests. Specifically, it covers the following tests:

• Navigate forwards to an empty, collapsed combobox in reading mode
• Navigate backwards to an empty, collapsed combobox in reading mode
• Navigate forwards to a filled in, collapsed combobox in reading mode
• Navigate backwards to a filled in, collapsed combobox in reading mode

In all cases, the included commands are:

• C (forwards) and Shift+C (backwards)
• E and Shift+E
• F and Shift+F
• Tab and Shift+Tab
• Down Arrow and Up Arrow (pressed twice to move the screen reader cursor across both the combobox and its label)

Current Screen Reader Behaviour

With their default settings, both JAWS and NVDA behave as follows:

• When moving the reading cursor to the combobox using quick navigation keys (C, E and F plus their Shift-modified counterparts), reading mode is retained.
• When moving the reading cursor to the combobox using the arrow keys (Down or Up Arrow as appropriate), interaction mode is automatically selected as soon as the combobox (not its labelling element) is encountered.
• When moving focus to the combobox with reading mode active using Tab or Shift+Tab, interaction mode is automatically selected as soon as focus lands on the combobox.

Questions/Comments

At a high level, there are two questions to answer for each of these outcomes:

1. Is the current behaviour felt to be expected or not?
2. Do we want to assert that the current behaviour should or should not be present?

Example expectations and outcomes:

• Quick navigation commands:
  • Expectation: screen readers should not switch from reading to interaction mode when navigating to the combobox using quick navigation keys.
  • Reasoning: it would reduce the convenience of navigating sequentially between multiple comboboxes/edit fields/form fields on a page, and could result in accidental editing of the combobox input value.
  • Proposed outcome: we assert that a mode switch does not occur for these commands, and the current behaviour of JAWS and NVDA would pass.
• Arrow keys:
  • Expectation: screen readers should not switch from reading to interaction mode when navigating to the combobox using the arrow keys.
  • Reasoning: the combobox control uses the arrow keys for movement into the listbox popup, but the user is starting in a reading context where they move the cursor. The sudden change in functionality feels jarring, and requires the user to explicitly determine how to leave the combobox control without inadvertently changing its value.
  • Proposed outcome: we assert that a mode switch does not occur for these commands, and the current behaviour of JAWS and NVDA would fail.
• Tab/Shift+Tab:
  • Expectation: screen readers should switch from reading to interaction mode when moving system focus to the combobox.
  • Reasoning: moving focus into the combobox indicates an explicit intent, either on the part of the user to move through interactive controls without explicitly reading the text content of the page, or on the part of a developer to place the user's focus in a useful control. Focus can be easily moved out of the control again, and having to explicitly switch modes before editing the combobox value or exploring the listbox popup may feel inconvenient/unnecessary.
  • Proposed outcome: we assert that a mode switch does occur for these commands, from reading to interaction, and the current behaviour of JAWS and NVDA would pass.

Implications

Within the ARIA-AT test format, it is not currently possible to assign command-specific assertions. In this case, we cannot assert within a single test that quick navigation and arrow keys should not result in a mode switch, but Tab and Shift+Tab should.

Practically speaking, this can only be resolved by either:

1. changing the test format and relevant tooling; or
2. separating all relevant tests into two.

Proposed Test Separation

As a reminder, here is the list of the four relevant tests in the plan:

• Navigate forwards to an empty, collapsed combobox in reading mode
• Navigate backwards to an empty, collapsed combobox in reading mode
• Navigate forwards to a filled in, collapsed combobox in reading mode
• Navigate backwards to a filled in, collapsed combobox in reading mode

This could be expanded to:

• Navigate forwards to an empty, collapsed combobox in reading mode (with Tab removed)
• Move focus forwards to an empty, collapsed combobox in reading mode (with Tab as the only command)
• Navigate backwards to an empty, collapsed combobox in reading mode (with Shift+Tab removed)
• Move focus backwards to an empty, collapsed combobox in reading mode (with Shift+Tab as the only command)
• Navigate forwards to a filled in, collapsed combobox in reading mode (with Tab removed)
• Move focus forwards to a filled in, collapsed combobox in reading mode (with Tab as the only command)
• Navigate backwards to a filled in, collapsed combobox in reading mode (with Shift+Tab removed)
• Move focus backwards to a filled in, collapsed combobox in reading mode (with Shift+Tab as the only command)

This doubles the number of tests in this group, and would expand the overall test plan from 76 tests to 80 (around a 5% increase).

Potential New Tests

The previous section covered moving to/ into a combobox, but not away from/out of it. Such tests are not included right now because they wouldn't exercise screen reader functionality relating to the combobox itself.

With mode selection added to the equation, we could add new tests to assert expected behaviour when moving away from a control that requires interaction mode to be active. Here is a list of some example tests, albeit in an abbreviated form:

• Navigate out of the start and end of an empty/filled in, collapsed/expanded combobox input (with Left/Right Arrow keys): eight tests.
• Move focus forwards and backwards out of an empty/filled in, collapsed/expanded combobox input (Tab/Shift+Tab): eight tests.
• Move focus forwards and backwards out of a listbox popup: two tests.

Note that:

• Not all of these test variations may be considered necessary. For example, do we need to evaluate behaviour when moving out of a filled combobox input as well as an empty one?
• To prevent this document becoming even longer than it already is, I haven't explored the current or expected behaviour of such tests here, nor the corresponding breakdown between tests, commands and assertions. Please keep in mind, as illustrated by the previous section, that this may not be straight forward.

Nevertheless, adding everything here on top of the test expansions from the previous section would expand the test plan from 76 to 90 tests in total, an increase of over 18%.

Modified Tests

Some existing tests in the plan could benefit from assertions being added related to mode selection. Again, these tasks are in an abbreviated form.

• Open an empty/filled in, collapsed/expanded combobox in reading mode: assert switch to interaction mode.
• Open an empty/filled in, collapsed/expanded combobox in interaction mode: assert interaction mode is retained.
• Narrow down the matching options in an empty/filled in, expanded combobox in interaction mode: assert interaction mode is retained.
• Close an empty/filled in combobox in reading mode: assert that reading mode is retained.
• Close an empty/filled in combobox in interaction mode: assert switch to reading mode?
• Navigate from an empty/filled in, collapsed/expanded combobox to the first/last option in a listbox popup in interaction mode: assert interaction mode is retained.
• Navigate to the first/last/next/previous option in a listbox popup in interaction mode: assert interaction mode is retained.
• Navigate out of a listbox popup by moving the editing cursor to the left/right/start of the textbox/end of the textbox in interaction mode: assert interaction mode is retained.
• Select an option in a listbox popup in interaction mode: assert interaction mode is retained.

Other Notes

• We must carefully weigh the impact of adding new tests to test plans, particularly if those new tests are quite repetitive, and/or when the test plan is long and complex already. The more tests exist in a plan with quite a similar feel, the more mentally tiring it is for human testers to complete the plan, and the higher the chances of mistakes.
• We also need to compare the pros and cons of adding new tests, vs adapting the test format and related tooling, vs doing nothing at this time.
• There are nuances around switching back to reading mode in some screen readers, based on the user's own mode selection history. For example, if an NVDA user manually switches to focus mode, moves focus to a combobox input, and then tabs to a link, browse mode will not become active. Evaluating these avenues of behaviour is not proposed here, and could be extremely complex to incorporate into a test plan.
• Thoughts are welcome on assertion and title wording, regardless of which approach(es) we do and do not take.

CC @IsaDC, @mcking65

[jscholes]

Idea: could an unexpected mode switch be listed as an unexpected behaviour?

[jugglinmike]

The Assistive Technology Automation Subgroup of ARIA-AT Community Group just discussed this issue. Meeting minutes are available on w3.org and also included below.

The full IRC log of that discussion

<jugglinmike> Matt_King: Ultimately, what we want to come out of this with is an approach to how we're going to test mode switching with JAWS and NVDA <jugglinmike> Matt_King: One question: Are they separate tests? <jugglinmike> Matt_King: And how do we build this into the plan? <jugglinmike> James_Scholes: This is quite long and raised questions about mode switching <jugglinmike> James_Scholes: If we have a test with, say, 5 commands, and we only want to include mode switching for one command, we can't do that without expanding the test into multiple tests <jugglinmike> James_Scholes: That raises concerns about the number of tests in a plan. If we were to add every new test that I proposed (which is far from a given), we'd be growing already-large Test Plan (from 76 tests to 90 tests) <jugglinmike> James_Scholes: And that makes it even more onerous for testers <jugglinmike> James_Scholes: There are still more questions. What is the scope of the mode switching functionality that we want to test? <jugglinmike> James_Scholes: Do we want to test only that mode switching occurs when we expect it to? Or do we also want to test that an unexpected mode switch does not occur? <jugglinmike> Matt_King: We could add a mode switch to the list of "undesirable behaviors" <jugglinmike> Matt_King: That way, we wouldn't have to constantly assert that it does not happen <jugglinmike> Matt_King: But it does feel weird to have an assertion like, "the mode did not change" <jugglinmike> James_Scholes: We're testing with defaults, but we feel that having the screen reader enter interaction mode in some cases could actually be quite harmful to screen reader users because it encourages accidents <jugglinmike> Matt_King: I think if we told them to not change the mode when you press the "down arrow" key, that might be going a bit beyond the scope of ARIA-AT <jugglinmike> James_Scholes: I'm not sure that there's a significant difference between asserting that something should happen and that something should NOT happen <jugglinmike> Matt_King: I've thought of the distinction being this: if you're a web developer, and you've coded something in a particular way using ARIA, what you need is a predictable experience. I guess that kind of moves into the usability space <jugglinmike> Matt_King: If people learn that there's some reading you can do that isn't reversible... <jugglinmike> Matt_King: Well, the more I think about this, I'm not so sure. <jugglinmike> James_Scholes: For me, it doesn't feel like we're crossing any line in the tests to require that the mode must be retained <jugglinmike> James_Scholes: Let's say you manually switch to interaction mode, then you tab into this combobox, and then you leave the combobox. In that case, you remain in interaction mode <jugglinmike> James_Scholes: I think asserting against *That* behavior would be going too far <jugglinmike> Matt_King: I think that the logical next step is to explore what would be the consequences and requirements (both at the test-writing level and the system level) to be able to add those two capabilities into the Test Plans <jugglinmike> Matt_King: I made a note to myself to add an issue and put it on next week's agenda <jugglinmike> Matt_King: I think you (James_Scholes) should go ahead with exploring it on the test-writing side <= Sam_Shaw [~Sam_Shaw@7e4e2622.public.cloak] has joined #aria-at <jugglinmike> Matt_King: But the work may go beyond what we can/should manage with a single issue <jugglinmike> James_Scholes: I'll think about this, we'll talk about it, and we'll talk about it more <jugglinmike> Matt_King: It may mean the abandonment of the CSV format <jugglinmike> Matt_King: We put off developing a tool for test composition in favor of just using Excel. We may need to revisit that decision <jugglinmike> James_Scholes: Perhaps

[mcking65]

@IsaDC @jscholes

Are there any unanswered questions remaining in this issue? I think we have addressed all of this with the V2 format and we could close this issue? Do you concur?

[IsaDC]

@IsaDC @jscholes

Are there any unanswered questions remaining in this issue? I think we have addressed all of this with the V2 format and we could close this issue? Do you concur?

Agree. We can close this issue.

[jscholes]

Yep, closing.