howard-e
commented
3 months ago @mcking65 it's been some time since we initially discussed this so wanted to bump this thread with some thoughts and questions and also follow up on if these requirements are still aligned given other changes.
One possible approach is that the commands file file name specifies the minimum version required. For example:
voiceOver_macOS+14.3-commands.csv
A potential alternative to creating a separate file could be adding a new column to the existing {at}-commands.csv file, and that's to primarily limit technical changes required in the script generator. There are pre-existing assumptions that a single commands.csv is matched to a single AT.
But in exploring this idea, I wonder if this negatively impacts test authors? A sole benefit of the above (to me) is working with already known files.
But a new column which could just indicate the minimumAtVersion would mean that additional rows would have to be created to store the version specific settings. I can acknowledge that the {at}-commands.csv files could get uncomfortably larger if there is a need to consistently handle more AT version specific setting changes with new AT version bumps.
Another thought is if a test author may want to remove commands for a specific AT version in the future. In this case, then a different column could be preferred over just minimumAtVersion which could handle version specific settings + command exclusions or just a new column along with minimumAtVersion which handles potential command exclusions but even more business logic in a single file could become problematic at some point.
Then, if a run in the test queue specifies either a minimum version of 14.3 or a version that is newer than 14.3, the test runner would use that commands file. Similarly, in the reports, the commands shown would match the version requirements specified by the commands file.
For backward compatibility, if there are multiple files for the same AT and one does not specify a minimum version, it would be assumed to be applicable to versions that are older than the version specified by the file with the earliest minimum version.
The version numbers specified would need to exactly match the version numbers specified in the version list in the app.
Sounds good!
To be completely robust, the version numbers specified in the app should have approximate release dates so versions can be reliably sequenced in time.
Sounds good and the app is already tracking those.
Some additional questions before moving this forward:
- Can we expect that a test author may also add and remove commands for a new version?
- What is the expectation of what’s displayed on the review pages when a version specific setting is present?
Something like the following in the commands instructions works?
... If any settings are specified in parentheses, ensure the settings are active before executing the command or command sequence.
- Space (virtual cursor active)
- Enter (virtual cursor active)
- Space ({NEW-JAWS-SETTING} for JAWS 2026)
Something like the following in the assertion tables works?
Space ({NEW-JAWS-SETTING} for JAWS 2026): 1 MUST, 0 SHOULD, AND 1 MAY assertions
Priority Assertion Phrase Assertion Statement MAY convey role 'alert' Role 'alert' is conveyed MUST convey text 'Hello' Text 'Hello' is conveyed
- Also any thoughts for the collection form in this repository's rendered pages?
- In the app, the collection form will already know which settings to show since the AT version requirement would be known. But the same question applies for the review page in the app.
We have now encountered a scenario where a new version of a screen reader changed its commands. PR #1020 added new instructions for quick nav settings. This means that there is now a minimum required version of macOS required for all VO test plans.
Going forward, we could encounter more scenarios like this. It could also be the case that a new version of an AT is released and we need the ability to run tests for both the older versions and the new version.
We could handle scenarios like this if the commands file specified the versions of an AT for which it is applicable. One possible approach is that the commands file file name specifies the minimum version required. For example:
voiceOver_macOS+14.3-commands.csvThen, if a run in the test queue specifies either a minimum version of 14.3 or a version that is newer than 14.3, the test runner would use that commands file. Similarly, in the reports, the commands shown would match the version requirements specified by the commands file.
For backward compatibility, if there are multiple files for the same AT and one does not specify a minimum version, it would be assumed to be applicable to versions that are older than the version specified by the file with the earliest minimum version.
The version numbers specified would need to exactly match the version numbers specified in the version list in the app.
To be completely robust, the version numbers specified in the app should have approximate release dates so versions can be reliably sequenced in time.