While adding support for AT version requirements in automation, I encountered a limitation on the specificity of macOS versions supported by Github runners that affects how we handle VoiceOver version requirements in automation. This issue is intended to bring this to the attention of the working group so that they can approve the proposed solution or offer an alternative.
Outline of issue:
Github Actions do not support specifying that a runner with a specific minor version is used for macOS workflows.
Only the major version can be specified and the actual minor version is not known until the job is completed.
Admins can specify a minor version in the required AT for a Test Plan Report. This makes version comparison to select a runner challenging.
For NVDA these version comparisons and selections are much more straightforward because the AT version isn't attached to the runner OS in the same way.
Example
If we were to continue to allow minor version requirements for VoiceOver, an example such as the following could occur:
Admin creates a new VoiceOver Test Plan Report with a minimum version of 13.0.1
The app can only specify that a workflow runs on a runner with a major version. The relevant major versions in this case are 13 or 14.
The only version that meets the minimum requirement of 13.0.1 is 14 even though it likely has less in common with 13.0.1 than version 13 does. Adding more complexity, Github could even be running 13.5 for jobs that specify version macOS-13 but we do not know this before starting the job.
The app has to decide whether automation supports this Test Plan Report. Since macOS 14 technically meets the criteria, the app offers automation as an option for running the Test Plan Run, and upon starting the collection job uses 14.
Proposed solution:
We limit the required and minimum AT version selection for VoiceOver to major versions. The bot can still report which minor version it used during the Test Plan Run.
Overview
While adding support for AT version requirements in automation, I encountered a limitation on the specificity of macOS versions supported by Github runners that affects how we handle VoiceOver version requirements in automation. This issue is intended to bring this to the attention of the working group so that they can approve the proposed solution or offer an alternative.
Outline of issue:
Example
If we were to continue to allow minor version requirements for VoiceOver, an example such as the following could occur:
macOS-13but we do not know this before starting the job.Proposed solution:
We limit the required and minimum AT version selection for VoiceOver to major versions. The bot can still report which minor version it used during the Test Plan Run.