search

w3c / aria-at

Assistive Technology ARIA Experience Assessment
https://aria-at.netlify.app
Other
154 stars 28 forks source link

Questions related to "setup test page" #77

Open spectranaut opened 4 years ago

spectranaut commented 4 years ago

Update on changes

In PR https://github.com/w3c/aria-at/pull/76, I added the new key setup_script_description to verifyATBehavior and I added a new column to the test spreadsheet @jongund and @yohta89 designed to record this string while test writing. I did this because Matt included the description of the script in a column (where at Jon and Yohta left it in a comment in the setupScript itself) so we needed to norm on one way or the other. Additionally, I added the setup test page script description to the test plans.

Question 1: Should we include this information in the test itsself, and how?

I tried putting it into the tests to motivate discussion, here is a menubar editor test as an example.

I think we should surface this summary to the testers themselves. Testers will need to be able to know when a setup script DID NOT work, and mark the test as "uncompletable" or "broken". If they are used to seeing a widget over and over again, they should be altered when the state should be different.

Question 2: Should we modify the way we describe the set up script to not be an imperative?

Right now, they all say "check the checkbox" or "open the submenu". Maybe we should change it to a description of the state after the script executes, like "the first checkbox will be checked" and "the submenu will be opened". From the testers perspective, I think the imperative will be confusing because they will be looking for imperative statements to execute.

mfairchild365 commented 4 years ago

Yes to both questions, and I like your answer to question 1.

ghost commented 4 years ago

To me tests became much clearer and less room for interpretation thanks to this!

mcking65 commented 4 years ago

I like this idea ... a lot! I would change the presentation a bit though. The instructions in the example currently say:

  1. Click the "Open test page" button below to open the example widget in a popup window
    • Setup test page script description: open 'Style/Color' menu item and move focus to the 'Italic' menu item checkbox option

If the test does not have a setup script, then item 1 above is fine except that it needs a period at the end, and I think we should use the word "test" instead of "example" and "activate" instead of "click":

  1. Activate the "Open test page" button below to open the test widget in a popup window.

And, if there is no setup script, I assume the nested list would not be present.

If a test does have a setup script, it might be more clear if we worded it as follows:

  1. Activate the "Open test page" button below, which will open the test widget in a popup window and automatically perform the following actions to prepare it for testing:
    • Open 'Style/Color' menu item and move focus to the 'Italic' menu item checkbox option

If it is worded this way, then we do not have to be as concerned about the voice of the setup script description. It also makes it very clear that activating the button has multiple effects.

spectranaut commented 4 years ago

I'll implement Matt's suggestion if no one has any further input!