Update test builder to support V2 of test format

mcking65 commented 1 year ago

Objective

Revise the aria-at build scripts so they support not only V1 but also the new V2 test format defined in #974.

While versions of the test plans that are written in the V1 format will all eventually be deprecated, for at least a few years, we will continue to need the ability to view plans written in the V1 format.

Note: Consider moving the directory tests/resources to scripts/resources or some other location outside of tests since the content of that folder is not related to any specific test plan and appears to be related to building the test plans.

Changes to build review files

When running the build, a review file for each test plan is generated in the directory build/review. When the builder is building a test plan that is written in the V2 format, it should generate an HTML file that has the following differences when compared to the files currently being generated for the V1 format.

This zip file contains a mockup of a portion of the slider test plan in V2 format. horizontal-slider.zip

Note that this description is referring to properties in support.json that are visible in the slider-refactor branch via #975. That branch includes a refactor of the horizontal-slider test plan into the V2 format.

Page title

Change the H1 to have format TITLE Test Plan (NUMBER_OF_TESTS tests) where TITLE is the value of the title refId in references.csv.:

Example:

<h1>Color Viewer Test plan (9 tests)</h1>

Filter radio group

Change the label of the radio group to communicate that the radios represent the AT covered by the plan:

Example:

<legend>Filter tests by covered assistive technology</legend>

To ensure there is a radio button for each AT that is covered, use the names of"AT_KEY-command.csv" files as the source of which AT are covered. For each command CSV file, include a radio button labeled using the corresponding name property in support.json.

Supporting documentation section

After the filters, add a level 2 heading with text "Supporting Documentation".

After the heading, list any of the following references that are included in references.csv. Since these apply to the entire test plan, do not include these references in the list of relevant specifications under the h2 for each test:

example
designPattern
developmentDocumentation

For each of the above refIds, include a link using link text from the linkText column and the URI from the value column.

Example:

<ul>
    <li><a href="https://www.w3.org/WAI/ARIA/apg/patterns/slider/examples/slider-color-viewer/"">APG Color Viewer Slider Example</a></li>
    <li><a href="https://www.w3.org/WAI/ARIA/apg/patterns/slider/">APG Slider Pattern</a></li>
    <li><a href="https://github.com/w3c/aria-at/issues/403">Developmen documented in ARIA-AT GitHub issue 403</a></li>
  </ul>

Changes in list under test title

Make the following changes in the ul under the heading with the test title, e.g., <h2 class="jaws nvda voiceover_macos">Test 1: Navigate forwards to a slider</h2>.

Remove specification of screen reader mode.

Example: <li>Mode: interaction</li>

Remove "applies to" statement.

Example: <li>Applies to: jaws, nvda, voiceover_macos</li>

Add a paragraph before the list with text from ariaSpecsPreface where ariaSpecsPreface is the value of the name property of an object in the testPlanStrings array in support.json and the paragraph text is the value of the string property in that object.

Change the content of the subsequent lis to:

Only include links for the refIds specified in assertions.csv for the assertions covered by this test as specified in the assertions column of tests.csv.
for each ARIA attribute covered by the test, make a link creating link text from the LinkText column and the URI from the value column of references.csv.

Example from slider test 1:

  <p>Tested ARIA features:</p>
  <ul>
    <li><a href="https://www.w3.org/TR/wai-aria#slider">slider role specification</a></li>
    <li><a href="https://www.w3.org/TR/wai-aria#aria-labelledby">aria-labelledby attribute specification</a></li>
    <li><a href="https://www.w3.org/TR/wai-aria#aria-valuenow">aria-valuenow attribute specification</a></li>
    <li><a href="https://www.w3.org/TR/wai-aria#aria-valuemin">aria-valuemin attribute specification</a></li>
    <li><a href="https://www.w3.org/TR/wai-aria#aria-valuemax">aria-valuemax attribute specification</a></li>
  </ul>

Changes under the `h3` for each AT

Make the following changes to the content that follows the level 3 heading for each AT, e.g., <h3>JAWS</h3>.

Remove the level 4 heading and content covering scripted instructions.

Example:

<h4>Scripted Instructions</h4>
        The following instructions are executed by a script in the test page to initialize the widget:
        <ol>
        <li>sets focus on a link before the Red slider</li>
      </ol>

Make the following changes under <h4>Tester Instructions</h4>:

Change the heading content to "Instructions".

Example: <h4>Instructions</h4>

Make the first instructions list item the instruction for configuring the AT with default settings. use the value of the defaultConfigurationInstructionsHTML property in support.json.

Example:

<li>Configure JAWS with default settings. For help, read <a href="https://github.com/w3c/aria-at/wiki/Configuring-Screen-Readers-for-Testing">Configuring Screen Readers for Testing</a>.</li>

Remove the instructions list item about setting reading or interaction mode.

Example: <li>Verify the PC Cursor is active by pressing Alt+Delete. If it is not, turn off the Virtual Cursor by pressing Insert+Z.</li>

Make the second instructions list item the open example instruction with form: `openExampleInstruction + " " + setupScriptDescription + "."`` where:

openExampleInstruction is the name of a string in array testPlanStrings in support.json.
setupScriptDescription is the value in the setupScriptDescription column in data/scripts.csv on the row where the setupScript value matches the setupScript value in the row in tests.csv for the test being presented.

Example:

<li>Activate the "Open test page" button, which opens the example to test in a new window and runs a script that sets focus on a link before the Red slider.</li>

Make the third instructions list item the perform task instruction with one of the following two forms:

If 1 or more commands assigned to this test in AT_KEY-commands.csv have settings specified, testInstructions + " " + commandListPreface + " " + commandListSettingsPreface + ":"
If 0 commands assigned to this test in AT_KEY-commands.csv have settings specified, testInstructions + " " + commandListPreface + ":"

Where:

testInstructions is pulled from data/tests.csv
commandListPreface is the name of a string in array testPlanStrings in support.json.
commandListSettingsPreface is the name of a string in array testPlanStrings in support.json.

This list item includes a nested list of all commands assigned to this test in AT_KEY-commands.csv where each list item contains a command-sequence string and an optional settings specification.:

Using values from commands.json, a string that represents the command. If the command is a sequence, each command in the sequence is separated by the string " then ".
If settings are specified for the command, they are included in parentheses. If there are 2 or more settings, settings after the first setting are prepended with the string " and ". Use the screenText property specified for each setting in the settings array for the AT in support.json.

The pairing of command-sequence string and settings specifications used in these list items is also used in other parts of the test plan display. For ease of reference, It is referred to as a command-specification string.

Example:

<li>
        Starting at the 'Navigate forwards from here' link, navigate to the 'Red' slider.
        Do this with each of the following commands or command sequences.
        If any settings are specified in parenthesies, ensure  the settings are active before executing the command or command sequence.
        <ul>
          <li>Down Arrow then Down Arrow (Virtual Cursor Active)</li>
          <li>Tab (Virtual Cursor Active)</li>
          <li>Tab (PC Cursor Active)</li>
        </ul>
      </li>

If any of the commands in the above list specify a setting, follow the instructions with a paragraph and ordered list for each setting that explains how to activate the setting.

The paragraph contains settingInstructionsPreface + " " + screenText + ":" where:

settingInstructionsPreface is the name of a string in array testPlanStrings in support.json.
screenText is the value of the screenText property for that setting in the settings array for that AT in support.json.

The ordered list contains the content of the instructions property for that setting in the settings array for that AT in support.json. The instructions property is an array, and each element in the array is an item in the ordered list.

Example:

    <p>To perform a command with virtual cursor active:</p>
    <ol>
      <li>Press Alt+Delete to determine which cursor is active.</li>
      <li>If the PC cursor is active, press Escape to activate the virtual cursor.</li>
    </ol>
    <p>To perform a command with PC cursor active:</p>
    <ol>
      <li>Press Alt+Delete to determine which cursor is active.</li>
      <li>If the virtual cursor is active, press Insert+z to disable the virtual cursor.</li>
    </ol>

Replace the current table listing assertions and their priorities with a series of headings and tables. One table for each command that is preceded by a level 4 heading. The order of the series is determined by the presentationNumber values in the AT_KEY-commands.csv file for the given test.

The heading has form:

command-specification string + ": X MUST, Y SHOULD, and Z MAY assertions"

Where:

command-specification string is defined above in the description of the instructions section.
X , Y, and Z represent the numbers of each level of assertion tested with this command.

The table has 3 columns:

Priority
Assertion Phrase
Assertion Statement

Where:

The values for each cell in the table are pulled from the tests.csv, assertions.csv, and AT_KEY-commands.csv files.
Rows are sorted first by priority descending and then assertion-sequence where assertion-sequence is the order of assertionId values in the assertions in the assertions column in tests.csv.
The table has an accessible label provided by the preceding level 4 heading.

Example from the JAWS section of the slider test 1 preview:

    <h4 id="t1-jaws-c1">Down Arrow then Down Arrow (virtual cursor active): 3 MUST, 3 SHOULD, AND 0 MAY assertions</h4>
    <table aria-labelledby="t1-jaws-c1">
      <tr>
        <th>Priority</th>
        <th>Assertion Phrase</th>
        <th>Assertion Statement</th>
      </tr>
      <tr>
        <td>MUST</td>
        <td>Convey role &#39;slider&#39;</td>
        <td>Role &#39;slider&#39; is conveyed</td>
      </tr>
      <tr>
        <td>MUST</td>
        <td>Convey name &#39;Red&#39;</td>
        <td>Name &#39;Red&#39; is conveyed</td>
      </tr>
      <tr>
        <td>MUST</td>
        <td>Convey value &#39;128&#39;</td>
        <td>Value &#39;128&#39; is conveyed</td>
      </tr>
      <tr>
        <td>SHOULD</td>
        <td>Convey Orientation &#39;horizontal&#39;</td>
        <td>Orientation &#39;horizontal&#39; is conveyed</td>
      </tr>
      <tr>
        <td>SHOULD</td>
        <td>Convey minimum value &#39;0&#39;</td>
        <td>Minimum value &#39;0&#39; is conveyed</td>
      </tr>
      <tr>
        <td>SHOULD</td>
        <td>Convey maximum value &#39;255&#39;</td>
        <td>Maximum value &#39;255&#39; is conveyed</td>
      </tr>
    </table>

CSV validation

Some validation rules to check for errors in the CSV content are documented in the Validation section of the Test Format V2.

mcking65 commented 1 year ago

@howard-e

Note that I have updated the V2 format specification with the following changes. I have not yet revised the requirements in this issue to reflect these updates.

Revised testId and assertionId descriptions:

Only include characters a-z, 0-9, and dash ("-").
Be generated from the test title by removing extraneous words, shortening some common words (e.g., navigate to nav), converting to lowercase, capitalizing the first letter of the second and subsequent words, and removing spaces and punctuation.
Be generated from the assertionStatement by removing extraneous words (e.g., 'conveyed'), shortening some common words, converting to lowercase, capitalizing the first letter of the second and subsequent words, and removing spaces and punctuation.

Revised presentationNumber specifications for tests and commands:

A positive number that controls the order of presentation of tests in the test runner and reports. By default, tests will be presented in numerical ascending order using the values in this column, i.e., the values are Number, not String, primatives when sorted.
A positive number that controls the order of presentation of commands in the test runner and reports. By default, commands within a test will be presented in numerical ascending order using the values in this column, i.e., the values are Number, not String, primatives when sorted.

Revised requirements related to refId values as described in the assertion and references sections:

The refId values designate references to the ARIA or HTML-AAM specification covered by the assertion. Typically, an assertion should cover only one ARIA or HTML feature. Some assertions will cover a behavior that is not described by any specification, so this cell will be blank in those rows.
The convention for ARIA attributes is that the refId is equivalent to the ARIA attribute, e.g., slider or aria-orientation. For HTML-AAM mappings, it is a good idea to avoid ambiguity by using a refId that includes the string html, e.g., htmlButton, or htmlLink.

Defined new values for reference types:

metadata: Indicates the refId is for information that applies to all tests in the plan.
aria: Indicates that the refId is for an ARIA attribute specification.
htmlAam: Indicates the refId is for an HTML element mapping specification.

Revised reference value specification for aria and htmlAam reference types.

Added requirements for linkText.

Added a section describing how link text and href values are calculated for reference links.

Updated validation rules.

howard-e commented 1 year ago

Noted, thanks for pointing that out

jscholes commented 1 year ago

Reopening, as merging #988 doesn't actually resolve this.

mcking65 commented 11 months ago

Using the preview content from #997and #998, I have Fully tested this against the specs. Since this work is already merged to master in aria-at and does not require a deployment, I am closing the issue as complete.

I have raised #1008, #1009, and #1010 to track required follow-up work and address problems I found during test.

w3c / aria-at