Prototype to automate ARIA AT tests in NVDA

[zcorpan]

I'd like to give the group a heads-up that I've started to work this week on a prototype to automate ARIA AT tests in NVDA.

[zcorpan]

Status update:

• Started a design document with goals and use cases.
• Met with NVDA developers. They are open to collaborating and interested in running automated aria-at tests in CI.
• Got the NVDA project set up locally on a Windows 10 laptop. Building NVDA works, running System Tests works.
• I've submitted a PR to nvda to experimentally add a new system test for the checkbox APG pattern. It works, but there's no actual integration with aria-at yet.
• At TPAC, we presented the ARIA AT automation protocol idea with the web-platform-tests team, and shared our vision of possibly sharing infrastructure between wpt and aria-at in the future, and standardizing a cross-AT automation protocol. They didn't seem opposed to the idea of expanding the scope of wpt to include testing ATs. See minutes.

For NVDA/aria-at integration, the idea is to have the tests in this repo be directly consumable for automation in NVDA's CI. This might need some changes to how tests are written/represented in this repository, since the steps and assertions are currently written to be understandable for a human, not necessarily for a script.

cc @feerrenrut , @WestonThayer

Adding Agenda+ to discuss in the telecon on Thursday.

[zcorpan]

Status update:

• I've updated the design doc with my approach with the test file format, authoring tests and running tests.
• I have proposed a test file format for ARIA AT automated tests (also described in the design doc). In the issue I also discuss that it's possible to augment the test files with expected test results, in cases of known test failures or known flaky tests in a particular AT.
• I have implemented a test runner using NVDA System Tests for that format, available for review on Bocoup's fork of nvda.
• For this to scale to all ARIA AT tests, we need to change the test source material to be able to generate these test files. This still needs discussion.

[mfairchild365]

@zcorpan I stumbled across this project which automates VoiceOver to a degree. https://github.com/ckundo/auto-vo

[WestonThayer]

I also discovered VoiceOver's AppleScript API via https://apple.stackexchange.com/questions/46521/how-do-i-find-out-the-applescript-commands-available-for-a-particular-app/46523#46523

MacOS does appear to support 3rd party speech synths, but I haven't discovered any API docs. Luckily VoiceOver's AppleScript API includes a way to get the speech string.

[jscholes]

Also potentially helpful: https://github.com/phenomnomnominal/screen-reader-reader/blob/fc55d1dd01862c11c148a28a4ee05fa79343baee/scripts/voiceover/get-voiceover-text.js#L10

[jscholes]

Well, more generally: https://github.com/phenomnomnominal/screen-reader-reader/

[zcorpan]

cc @s3ththompson @jugglinmike @mzgoddard