search

w3c / aria-at-app

Test runner and results reporter for ARIA-AT
http://aria-at.w3.org/
Other
35 stars 15 forks source link

Test runner UI cannot accurately describe the state of assertion verdicts #1045

Open jugglinmike opened 5 months ago

jugglinmike commented 5 months ago

Currently, the "Test Plan Run" page presents Testers uses a two-state checkbox to convey the state of each assertion verdict, e.g.

Which statements are true about the response to f (browse mode on)?

  • [ ] Role 'group' is conveyed

The value of the assertion verdict is actually indeterminate in two cases, making a two-state checkbox insufficient:

  1. When human Testers begin manually running a test and have not yet reported whether or not the assertions are satisfied.
  2. When a "bot" tester has collected AT responses for the Test Plan Run and a Test Admin has re-assigned the Test Plan Run to a human Tester

In effect, the "unchecked" state ambiguously describes two different states: "this assertion is not satisfied," and "this assertion has not yet been evaluated."

To date, this ambiguity has been tolerable because the human Tester is the sole arbiter of assertion verdicts. They tacitly track their own progress through the Test Plan Run.

However, the ambiguity will become more confusing as the system gains the ability to automatically re-use verdicts when it recognizes historic test results. In those cases, a human Tester who receives a Test Plan Run from a Bot will not be able to recognize which assertions have been recognized as "failed" and which have not been automatically judged by the system.

jugglinmike commented 4 months ago

The community group discussed this issue briefly during today's meeting.

mcking65 commented 3 months ago

Please implement using Yes/No radios like we have for undesirable behaviors. This will give all the questions the same format.

If neither radio is checked, the verdict is not assigned. On submit, make sure all verdicts are assigned.

For accessibility, ensure that each assertion is a radio group that is labeled by the assertion. Also, please don't use a table element in the layout of the radio buttons and assertions.

jugglinmike commented 3 months ago

Thanks, @mcking65!