Mockups for Test Plan Management, Test Queue, and Candidate Test Plans

[isaacdurazo]

Test Plan Management, Test Queue, and Candidate Test Plans Mockups

Test Plan Management - Text-version Mockup

From this new page, Admins can control all aspects of Test Plans, meaning some features have moved from other pages to this one.

Manage Assistive Technology Versions

This Disclosure component was previously displayed in the Test Queue. It's now been moved to this page.

Add Test Plan and Assistive Technology combinations

Previously displayed in the Test Queue as "Add Test Plans to the Test queue", this Disclosure component has now been moved to this page.

Status Summary table

This table displays all Test Plans currently added to the ARIA AT App. From here, Admins can control their Phase from a high level, meaning a Test Plan and all its AT and browser combinations are moved from one phase to another at once. The details of this table are these:

• Test Plans column: Lists all Test Plans that have been added to the ARIA AT App
• Phase column: Lists a dropdown for each test plan to modify its Phase: Draft, Candidate, and Recommended
• Status column: Displays the high-level status in which a Test Plan is. Depending on the Phase a Test plan is in, the status could be different. Here are the details. Please note some scenarios could be missing.
  • Draft:
    • 0 Testing Rounds Completed
    • X of Y Testing Rounds Completed
  • Candidate
    • 0 Reviews Completed
    • X of Y ReviewsCompleted
  • Recommended
    • Approved

Assistive Technology Table

This table displays all Test Plans under a particular AT and the different Browsers in which these are being tested. The Test Plan Management page includes multiple Assistive Technology Tables, one for each Assistive Technology recognized by ARIA-AT.

• Test Plans column: Every cell in this column has a Test Plan name, the phase in which it is next to it, and the published date underneath. There's also a plain text "Update" link (which should be a button but wanted to make it look less prominent) next to the date. The Publish date and Update buttons have been moved from the Test Queue to this page.
• Browser and Status column: There could be more than one of these columns depending on how many Browsers we have available. This column displays the status in which a particular Test Plan being run with a particular AT and Browser combination is The statuses depend on the phase and could be the following:
  • Draft:
    • Testing Round in Progress
    • Testing Round Completed
  • Candidate
    • Review in Progress
    • Review Completed
  • Recommended
    • Approved

Test Plan Management - Visual Mockup

Test Queue - Text-version Mockup

This page is now primarily for Testers, with all Admin actions moved to the Test Plan Management page. A few more changes have been made to improve the UX. Here are the details:

1. "Manage AT Version" and "Add Test Plans to the Test Queue" disclosures have been moved to the new Test Plan Management page.
2. Details of the new AT/Browser table
   • Test Plan Column:
     • Displays the name of the Test Plan as a link if the user is assigned or as plain text if they are not
     • The Phase is displayed next to the Test Plan name. We used to have a dedicated column called "Report Status" for this, but using the word status is confusing. There's really no need to have a column for this anymore because we don't control it from here but from the Test Plan Management page.
     • "Open Run as...", "Delete Test Results for..." and "Remove Test Plan", have been moved to be displayed under the Test Plan name. For sighted users, we can have these hidden by default and appear on hover, which could make the UI look cleaner.
   • Testing Status. There are different statuses a Test Plan could be in. A Test plan can have multiple statuses at once. These statuses pertain only to "testing" and not to other kinds such as "reviewing". Here are the details:
     • No Testing Round started yet
     • X Testing Rounds in progress
     • Testing Round Completed
     • X Testing Rounds Completed
     • X Conflicts
   • Testers. This column doesn't have any changes

Test Queue - Visual Mockup

Candidate Test Plans - Text-version Mockup

This page hasn't changed much. Here are the details:

1. The "Review Status Summary" table has been moved to the top so the user can take a look at a high level first and be consistent with the similar summary table we have on the Test Plan Management page
2. The "Mark as..." button has been removed. That Admin action is now on the Test Plan Management page.

Candidate Test Plans - Visual Mockup

Afterthoughts

These new mockups represent major changes to the ARIA AT App and the Working mode. There are still some questions that remain unanswered that should be discussed. Here a few I was running into while I worked on these mockups with Seth last year:

1. What happens in the following scenario? I add Disclosure Nav Example with JAWS. It then moves from Draft to Candidate, then later I add Disclosure Nav Example with VO, which theoretically should start as a Draft. Would both Test Plans be in different phases? They can't. So in a way, the Test Plan review process should be like a train leaving the station. after the first group of ATs is published as Recommended, any additional ATs that we add should start as recommended too.

2. What are the minimum requirements to move a test from "draft to candidate"? Do we need to specify an X number of finished testing rounds with no conflicts?

3. What happens if we need to update a Test Plan that has been moved to recommended? Should it go back to Draft?

4. Similarly, what about the scenario where we might need to add more test results to a Test Plan that has moved to Candidate or Recommended?

There are probably more things I'm not considering, hence I think a broad discussion should be had about these mockups to see how we can address these questions and their impact.

[jugglinmike]

Great work, @isaacdurazo! As always, your attention to detail for design and documentation is inspiring :)

In each row of the "Status Summary" table, a phase name is displayed twice: once in the "Test Plans" column and also in the "Phase" column (via the active state of the <select> element). I initially thought they were redundant until I noticed that they differ in the final row of the visual mockup. Since your written description doesn't mention that a Phase title appears in the "Test Plans" column, I'm wondering what this is about.

• Draft:
  • 0 Testing Rounds Completed
  • X of Y Testing Rounds Completed
• Candidate
  • 0 Reviews Completed
  • X of Y ReviewsCompleted

For each of these, when would the first option be used instead of the second?

• Test Plan Column:
  • Displays the name of the Test Plan as a link if the user is assigned or as plain text if they are not

This is a little bit subtle for my liking. I would prefer to see a dedicated link whose text described the action and for that link to be disabled when viewed by unassigned Testers. That said, the density of information in this area might justify a more concise interface like this.

More broadly, it seems to me that this table is trying to serve the needs of both Test Admins and Testers, and that as a result, it feels somewhat overcomplicated for each of them individually. I haven't been participating in design reviews, so I'm wondering if you folks have previously considered presenting different interfaces based on the role of the viewer.

[isaacdurazo]

Thanks for your feedback @jugglinmike. I think I got some things wrong here that I need to correct.

1. You are right about the redundancy in the "Status Summary" table. There's really no need to have it twice. That last row where the phases differ is actually an overlook on my end.
2. The difference between "0 Testing Rounds Completed" and "X of Y Testing Rounds Completed" is that the first one has no one assigned yet, hence no progress has been made. The second one is for when 1 or more testers are assigned without necessarily having made progress yet. I'm now thinking we might want to reword these. I do see now how they can be confusing.
3. I'm just realizing Test Plans should not be linked from here. If the Admin wants to open one of them, they should probably navigate to the Test Queue to use the "Open Run as..." or assign themselves as testers to do so.

[mcking65]

@isaacdurazo

Thank you for the proposal. I echo Mike's appreciation of detail. The details provide the food we need for discussion.

There is a lot to discuss. I want to first focus on some of the higher level concepts before we talk about how to refine the design.

Conceptually, the information presented on the "test plan management" page is about much more than test plans. There is a lot of data and information that is not part of test plans. In our discussions with Seth, I think I proposed we call it the "Data Management" page.

Feedback related to "Status Summary table"

This table displays all Test Plans currently added to the ARIA AT App. From here, Admins can control their Phase from a high level, meaning a Test Plan and all its AT and browser combinations are moved from one phase to another at once.

Note that browsers are not part of test plans. See the definition of "test plan" in the glossary.

From the glossary:

A test plan includes all tests for a particular implementation of an ARIA design pattern. A plan covers all AT currently supported by the project, e.g., all tests for all AT for the grouped checkbox example.

There is one correction that I need to make in the above definition. In the future, we will have AT in the system that are not part of a specific test plan. A test plan has a list of "in-scope AT". Those are the AT for which commands are provided in the tests that are included in the plan.

The important thing to note here is that a test plan does not include any browser information. It does not include any data that results from running a test. The test plan is only the tests, assertions, and AT commands for a given test case. Basically, the test plan is the specification of AT behaviors for a given accessibility semantics usage pattern. To put it in wpt.fyi terms, it would be like the normative requirements provided by the HTML spec for a given element or attribute.

Think of the working mode as the equivalent of the W3C process for spec development. We just want stakeholders to agree on requirements. For aria-at, that means we are just trying to get stakeholders to agree on what the AT expectations are for a certain ARIA usage pattern. When they agree, the plan is recommended for use, and we can use it to test the in-scope AT with any browser.

My first observation about the table is that it should probably be titled "Test Plans" to indicate that it lists all the test plans in the system.

My second observation is that we need some additional information about the plans.

• Current version (We currently show this in the test queue, and I believe we are using the date/time the plan was merged to main)
• List of in-scope AT. Note this is just the names of the AT, not their versions because the plan is AT version agnostic.

Before talking about the "status" column, we need to talk about the purpose of running tests as it relates to the working mode.

There is only one reason for running tests before a test plan is recommended: Proof the plan.

Imagine a world where we could write a test plan kind of like a novel. We would read and edit our draft until we like it. We'd give the candidate draft to the publisher who would twist it around until we secretly dislike him but agree he's wonderful. Then, the publisher would recommend it to the world. We could then use the recommended test plan to test any in-scope AT with any browser to produce awesome interoperability reports.

Unfortunately, that won't work in practice. We run the draft and candidate plans to generate test results because:

• The test run results help us find errors in the plans before we talk to AT developers. The results help us make sure that human testers are able to interpret the tests consistently.
• When talking with AT developers, the AT developers can look at the run results to spot problems. If their AT does not satisfy an assertion, maybe that indicates that the plan requires behaviors that they don't agree should be required.

It's important to note though, that the run results (per the glossary) are not part of the plan. Per the working mode, we require a certain amount of "proofing" of a plan for it to move out of draft or for revisions of a candidate plan to be accepted. The "proofing" is simply run results that people agree are complete and accurate.

The remaining columns in the table, whether they are called status or something else, should give admins the info they need to know to make decisions about next steps for the plan, including whether to change its phase. We need to do a bit more thinking on this part of the design.

Note that important events can arise during the recommended phase, and the columns in this table should help us triage them as well.

In Draft, the requirement for moving to candidate is that every in-scope AT has a complete set of results for at least one browser. A "Complete" set of results is 2 or more testers executed the same test plan run and their results agree. So, for a draft plan, it will be important to see whether these requirements are met, and if not, why not.

Feedback on "Assistive Technology Table"

I could be overlooking something here, but the benefits of moving functions for managing test plan run results from the test queue page to this data management page are not apparent to me. It feels like a lot of work, and I am not confident it would improve usability. For some of the admin functions, it seems like it would reduce usability because we'd be doing more page and context switching.

Another thought I have related to this part of the proposal is that we need to spec out the status values for test plan run results. I don't think there are so many. The states that I believe we have are:

• Not started
• X% complete with y conflictsand z open issues
• "Finalized" or "Published"

I think Those are the only states.

Percent complete = number of tests with results from all assigned testers / number assigned testers * number of tests

When testing is 100% complete and there are 0 conflicts and 0 open issues, the results can be "Finalized" or "Published". I believe there is currently confusion around this state and it deserves more discussion. Recall that for a test plan to move from draft to candidate, the test admin is responsible for ensuring we have "complete" or "finalized" test results for at least one browser for each in-scope AT.

[isaacdurazo]

Hi folks,

here are the updated mockups and a description of the changes based on our last conversation about this issue. Please note that I'm describing the changes based on what's currently implemented on ARIA-AT app and not based on the last version of these mockups

Test Management

Changes

1. Change the page title to "Data Management"
2. Status Summary table
   • "Versions" column (new). In each cell, under this column, we display the following information:
     • Current Version: E.g. Apr 13, 2023 at 4:22:06 pm UTC - Update cg and wg references in readme (1bb257e)
     • Update TestPlan/Version button: this used to be in the Test Queue
     • Prior Version: Apr 10, 2023 at 6:22:22 pm UTC - Remove Tab and Shift+Tab from radiogroup tests when navigating out of the start and end of a radio group (reading mode and VoiceOver only) (#928) (1768070)
     • See all Versions button: This is a new feature. I'm imagining this could be a popup that displays the list of all versions.
   • "Status" column (new): each cell in this column displays information about the date for when test plans were at each phase. Under this same column, I am imagining we could also display whatever conditions have been met so the Admin has the necessary info to move a Test plan from one phase to another. Please note that this last piece of information is not in the Visual Mockup.
   • "In Scope ATs" column (new): Each cell in this column displays the ATs in scope for a particular Test Plan. E.g. "JAWS and VoiceOver for Mac"

Visual Mockup

Test Queue

Changes

1. Remove the "Manage AT Versions" and "Add Test Plans to the Test Queue" disclosures. These disclosures are currently under Data Management and they don't need to be here as well. This is an Admin functionality
2. Remove the "Update Test Plan" button. This feature should be under Data Management. Also, this should be used from the Test Plan's highest level view, meaning not for all Test Plans under every AT/Browser.
3. Test Plan column:
   • Move the "Open run as..", "Delete Test Results for...", and "Remove Test Plan" buttons under each Test Plan name in this column. These buttons used to be under Actions.
   • Move the "Phase" label to be next to each Test Plan name under this column
4. Report Status column:
   • Remove the "Phase" label. This now is under the "Test Plan" column
   • Remove the button that modifies the phase of a Test Plan. This functionality should now be under Data Management.
   • Update styling of status messages based on Visual Mockups'
5. "Actions" column: This column can now be removed since these buttons have been moved to the "Test Plan" column

Visual Mockup

Candidate Test Plans

Changes

1. Move the Summary table to the top. This is so users can have a high-level view first.
2. Update styling of status messages based on Visual Mockups so they are consistent with the Test Queue
3. Remove the button that modifies the phase of a Test Plan. This is now under the Data Management page
4. Move info about the Candidate Phase start date and Target competition to Summary Table. This info is for a Test Plan alone, not for a Test Plan being run under a particular AT/Browser.

Visual Mockup

[jscholes]

@isaacdurazo Thanks for these updates! After an initial scan, the primary concern that stands out to me is how many items are being crammed into single tabular cells.

As you may be aware, screen readers provide commands to traverse tables by rows and columns, e.g. to move down a specific single column to consume information. This is already quite difficult in parts of the ARIA-AT app including the test queue, because of:

• cells having both data and controls within them;
• the long, quite-unwieldy format of some of the data, like test plan versions that are made up of a data, commit hash and commit message; and
• semantics being used, like lists, to group multiple pieces of information within a cell.

Additionally, screen readers will announce row/column headers when changing between relevant cells. The more information is placed in a row header cell, the harder the page is to use, because that information will be spoken each time the active row is changed. From this point of view, the most concerning change is in the Test Queue:

3. Test Plan column:
   • Move the "Open run as..", "Delete Test Results for...", and "Remove Test Plan" buttons under each Test Plan name in this column. These buttons used to be under Actions.
   • Move the "Phase" label to be next to each Test Plan name under this column

This is a huge amount of information for a screen reader to speak when switching between rows of a table in the queue, which is an action that is frequently carried out. I would go as far as to say that it will make the table unusable, without disabling the header functionality built into the screen reader, meaning that no information about the plan will be spoken at all unless the user happens to be positioned in the first column.

Similar concerns are raised by:

• Test Management: the "Versions" column in the Status Summary table listing two versions of a plan. As highlighted already, one version is quite hard to read in a cell.
• Test Management: the "Status" column potentially containing multiple bits of data in each cell, making it harder to scan.
• Test Management: the "In Scope ATs" column containing grouped info, i.e. multiple ATs.

[ccanash]

Hi @mcking65, for Candidate Test Plans can we consider the mockup approved? For the test queue @jscholes raised a concern about the accessibility for screen reader. We can perhaps discuss this during our call on Wednesday as similar concerns appear for #648