search

w3c / aria-at-app

Test runner and results reporter for ARIA-AT
http://aria-at.w3.org/
Other
35 stars 15 forks source link

Incorrect VoiceOver Bot Results for Color Viewer Slider Test Plan #1136

Open IsaDC opened 1 week ago

IsaDC commented 1 week ago

The results recorded by the VoiceOver bot are not valid, except for Test 8: Decrement a slider to the minimum value. I haven't been able to reproduce the bot's results manually, but there is a fresh run in the Sandbox for you to take a look.

Example of My Results Compared to the Bot's:

Test 1: Navigate forwards to a slider

After 'Control+Option+Right Arrow then Control+Option+Right Arrow':

IsaDC commented 1 week ago

CC @mcking65 @jugglinmike

css-meeting-bot commented 1 week ago

The ARIA-AT Community Group just discussed w3c/aria-at-app#1136: Incorrect VoiceOver Bot Results for Color Viewer Slider Test Plan.

The full IRC log of that discussion <jugglinmike> Topic: w3c/aria-at-app#1136: Incorrect VoiceOver Bot Results for Color Viewer Slider Test Plan
<jugglinmike> github: https://github.com/w3c/aria-at-app/issues/1136
<jugglinmike> IsaDC: Basically, the results from the bot don't match
<jugglinmike> James_Scholes: Is this happening on all of the tests?
<jugglinmike> IsaDC: There is only one test out of nine that the results match
<jugglinmike> James_Scholes: In the test that matches, does it match for all commands?
<jugglinmike> IsaDC: There's only one command
<jugglinmike> Hadi: I was just in the Test Queue. Just before Matt assigned me to Color Slider, I had a different test in the "main" region. Now, it has changed. Is it expected that I can see only one at a time?
<jugglinmike> Matt_King: There is only one Test Plan Run assigned to you, now--the color viewer slider. Can you find it?
<jugglinmike> Hadi: Yes, I can find it. This is much cleaner
<jugglinmike> jugglinmike: I don't know what's going wrong here, but the issue gives us everything we need to start investigating. Thank you!
<jugglinmike> s/wrong here/wrong in this issue, IsaDC/
<jugglinmike> Zakim, end the meeting
jugglinmike commented 2 days ago

Thanks for the report, @IsaDC! I can't reproduce this on macOS 13, but that's of limited relevance since both you and the bot are using macOS 14. Someone on my team will be investigating with the latest release soon.

For what it's worth, here's what I observed on macOS 13 Red 128 Red slider You are currently on a slider. To start interacting with the slider, press Control -Option -Shift -Down Arrow. Or, with the insignificant empty spaces trimmed: > Red 128 Red slider You are currently on a slider. To start interacting with the slider, press Control-Option-Shift-Down Arrow.
jugglinmike commented 2 days ago

By the way, here's the command I used to run the tests:

node aria-at-automation-harness/bin/host.js run-plan \
  --plan-workingdir aria-at/build/tests/horizontal-slider \
  '{reference/**,test-*-voiceover_macos.*}' \
  --agent-web-driver-url=http://127.0.0.1:4444 \
  --agent-at-driver-url=ws://127.0.0.1:4382/session \
  --reference-hostname=127.0.0.1 \
  --agent-web-driver-browser=safari