w3c / aria-at

Assistive Technology ARIA Experience Assessment
https://aria-at.netlify.app
Other
152 stars 27 forks source link

Impact of hint text/help messages on test results #365

Open jscholes opened 3 years ago

jscholes commented 3 years ago

This came up on an ARIA-AT CG call recently. For some controls, JAWS and macOS VoiceOver speak some associated help/hint text, which in VoiceOver's case is often quite substantial. Help text announcements are enabled by default, so will be heard by human testers. The question is, should these messages contribute to the passing or failing of assertions in any way?

A practical example: on the Mac, the main announcement for an editable combobox doesn't convey editable state, but the hint text does. If we provide testers with an assertion that the editable state was conveyed, should it pass or fail? Note that AFAIK, the help text isn't repeated when querying information about the control e.g. with VO+F3, so the assertion will fail in at least some circumstances.

jscholes commented 3 years ago

Update following Feburary 4th community group meeting: we ideally need more examples of where hint text from a screen reader conveys information considered critical to an assertion passing. In the case of macOS behaviour for combo boxes:

jscholes commented 3 years ago

Comment from @robfentress:

One issue I had when I was doing testing before was that I wasn't sure what to include when recording the output for VoiceOver. In particular, in the default configuration, VO announces hint text about the item in the VO cursor after a brief delay. I don't remember if we ever decided how to handle this. Would it simplify things if we changed the configuration instructions to instruct testers to turn off hints in the VO Configuration Utlity before conducting testing? Is any information we are looking for in our assertions communicated exclusively through hint text? If hints are to remain enabled in our configuration recommendations, then I think we should probably add something to our instructions telling users to make sure to wait long enough for them to surface.