Update VoiceOver Utility to allow VoiceOver to be controlled with AppleScript

[cmorten]

Tool name

VoiceOver Utility

Tool license

Copyright © 2005-2017 Apple Inc. / Unknown

Add or update?

• [ ] Add
• [X] Update

Desired version

Version 10 / All

Approximate size

No response

Brief description of tool

VoiceOver is a screen reader system application that provides auditory descriptions of elements help you easily navigate your screen with keyboard or gestures.

You can configure VoiceOver through the system application VoiceOver Utility.app. One such option is a checkbox Allow VoiceOver to be controlled with AppleScript which allows the user to use AppleScript scripts to automate VoiceOver tasks. (Reference: https://support.apple.com/en-gb/guide/voiceover/cpvougen/mac).

This feature request is to enable this option for the macos virtual environments.

URL for tool's homepage

https://support.apple.com/en-gb/guide/voiceover/welcome/mac

Provide a basic test case to validate the tool's functionality.

#!/bin/bash

test -f "/private/var/db/Accessibility/.VoiceOverAppleScriptEnabled" && echo "VoiceOver AppleScript Control Enabled"

Virtual environments affected

• [ ] Ubuntu 18.04
• [ ] Ubuntu 20.04
• [X] macOS 10.15
• [X] macOS 11
• [ ] Windows Server 2016
• [ ] Windows Server 2019
• [ ] Windows Server 2022

Can this tool be installed during the build?

I'm currently uncertain if you can set this option programatially...

From what I can tell, whether this is enabled or not is driven by the presence of a /private/var/db/Accessibility/.VoiceOverAppleScriptEnabled file (containing the single character a - as determined by digging into VoiceOver Utility / ScreenReaderCore.framework), but this directory has SIP so writing the file I'm assuming is a no-go (disabling SIP not an option).

It also appears trying to import a .voprefs file with the SCREnableAppleScript key set to true is simply ignored.

Attempts to UI script the clicking of the checkbox appears to just being doing nothing in GitHub actions for macos-11 or macos-10.15 envs, and screenshots of the final state look like nothing has happened at all (checkbox remains unchecked, no sign of SecurityAgent popup window etc.). Attempts locally (on Monterey) work fine though require password entry through the SecurityAgent dialog. Curiously starting VoiceOver Utitlity using sudo results in similar behaviour to what is observed (afaik) on the virtual envs. i.e.

sudo /System/Applications/Utilities/VoiceOver\ Utility.app/Contents/MacOS/VoiceOver\ Utility 2&> /dev/null &

Would be very curious to know if there is anything that can be done to have this option enabled for the virtual envs. Atm struggling to see a way past other than disable SIP, enable the option by adding the file to the protected directory, and then re-enable SIP. I suspect this isn't an option for security reasons + faff of trying to deal with the recovery mode etc.

Tool installation time in runtime

No response

Are you willing to submit a PR?

No response

[mikhailkoliada]

@cmorten Hello!

Thanks, we will take a look at it, the best thing to do this is probably using applescript.

[cmorten]

As a starter for 10, here’s what have tried so far in GitHub actions…

#!/bin/bash

# Debug screenshots
mkdir -p ./screenshots

screencapture ./screenshots/voiceover_utility_started.png

osascript ./.github/workflows/enableVoiceOver.applescript

if [ ! -f "/private/var/db/Accessibility/.VoiceOverAppleScriptEnabled" ]; then
  echo "Unable to enable AppleScript control for VoiceOver"
  exit 1
else
  echo "Successfully enabled AppleScript control for VoiceOver"
  echo "cat /private/var/db/Accessibility/.VoiceOverAppleScriptEnabled"
  cat /private/var/db/Accessibility/.VoiceOverAppleScriptEnabled
fi

delay 1

log "Activating VoiceOver Utility..."
tell application "VoiceOver Utility" 
    activate
end tell

delay 1

tell application "System Events"
    repeat while not (exists window 1 of application process "VoiceOver Utility")
        delay 0.1
    end repeat

    do shell script "screencapture ./screenshots/voiceover_utility_activated.png"

    log "Ticking checkbox..."

    tell application process "VoiceOver Utility"
        repeat until (exists checkbox 2 of splitter group 1 of window 1)
            delay 0.1
        end repeat

        click checkbox 2 of splitter group 1 of window 1
    end tell

    delay 1

    do shell script "screencapture ./screenshots/voiceover_utility_checked.png"
end tell

log "Quiting VoiceOver Utility..."
tell application "VoiceOver Utility"
  quit
end tell

Replacing checkbox 2 with checkbox 1 works fine for toggling the "display welcome…" checkbox, but the script as stands is lacking in some way for the AppleScript checkbox 🤔

[cmorten]

Struggling to find a workable solution, SIP seems to be a real blocker!

I’m aware that for security reasons SIP is not being disabled for images, but for https://github.com/actions/virtual-environments/issues/650 was temporarily disabling for building the image considered? E.g.

Template:

        {
            "type": "shell",
            "execute_command": "chmod +x {{ .Path }}; sudo {{ .Vars }} {{ .Path }}",
            "script": "./provision/core/reboot-recovery.sh",
            "expect_disconnect": true
        },
        // … SIP disable, limited changes (e.g. only changes a user could instrument through an Apple app like System Preferences), and SIP re-enable. Then we reboot back.
        {
            "type": "shell",
            "execute_command": "chmod +x {{ .Path }}; sudo {{ .Vars }} {{ .Path }}",
            "script": "./provision/core/sip-actions.sh"
        },
        {
            "type": "shell",
            "execute_command": "chmod +x {{ .Path }}; sudo {{ .Vars }} {{ .Path }}",
            "script": "./provision/core/reboot.sh",
            "expect_disconnect": true
        },

reboot-recovery.sh:

#!/bin/bash -e -o pipefail

sudo nvram internet-recovery-mode=RecoveryModeDisk

sudo reboot

Where can be some subtle differences require depending on OS version. Ref: https://apple.stackexchange.com/questions/367336/can-i-initiate-a-macos-restart-to-recovery-mode-solely-from-the-command-line

sip-actions.sh:

#!/bin/bash -e -o pipefail

# Disable SIP
csrutil disable

# Enable VoiceOver AppleScript control (Voicer Utility > General > Allow VoiceOver to be controlled by AppleScript)
echo -n "a" > /private/var/db/Accessibility/.VoiceOverAppleScriptEnabled

# Enable SIP
csrutil enable

# Ensure reboot normally
nvram -d recovery-boot-mode

[miketimofeev]

@cmorten we haven't had a chance to take a look at the script so far yet we enabled the setting in our templates. The next deployed images will have this option enabled. The deployment is starting on January 10 and will take about 3 days.

[cmorten]

@miketimofeev thanks for your time on this!

It appears I missed a piece of the puzzle, which is the setting of:

defaults write com.apple.VoiceOver4/default SCREnableAppleScript 1

which unlike the file, doesn't require any SIP negotiation. Starting to doubt if there are other elements that have been missed...

[ckundo]

happy to contribute here where needed. would love to use this feature to run https://github.com/AccessLint/voiceover.js in CI

[coryrylan]

+1 happy to help where I can. I attempted to get around this with similar strategies without success

https://github.com/coryrylan/web-test-runner-voiceover/blob/main/src/voiceover/settings.ts#L39 https://github.com/coryrylan/web-test-runner-voiceover/blob/main/src/voiceover/system.ts#L44

[cmorten]

Delighted to say this appears to be working (at least for simple examples!). I've run a simple experiment and have observed the following on the macos-11 agent:

VoiceOver Utility shows the AppleScript control checkbox to be checked:

I am able to start VoiceOver programatically:

I am able to use AppleScript to move the vo cursor (from the desktop to the first item of the dock):

🎉

(Ref: https://github.com/guidepup/guidepup/actions/runs/1765474785)