search

w3c / aria-at-app

Test runner and results reporter for ARIA-AT
http://aria-at.w3.org/
Other
35 stars 15 forks source link

Bot Run Assignment Failed #1226

Open IsaDC opened 6 days ago

IsaDC commented 6 days ago

Steps to Reproduce:

  1. Navigate to the Test Queue and select the "Navigation Menu" button (V24.09.19).
  2. For either NVDA or VoiceOver, press the "Assign Testers" button and select the bot.

Expected Behavior:

The bot should be assigned to the selected test run.

Actual Behavior:

After the "Assign Testers" dialog is displayed, the bot is not assigned to any test run.

Alternate Procedure:

Tried adding the test plan from the "Data Management" page and selecting "Add Bot Run" for both NVDA and VoiceOver. This method also fails.

Additional Information:

Both bots run successfully in the sandbox environment, indicating that the issue seems specific to the test queue.

Suggested Solution:

Fixing this issue as soon as possible would be greatly appreciated, as it significantly speeds up the testing process.

IsaDC commented 11 hours ago

Update:

The bots are now being assigned correctly; however, the test run is immediately canceled after assignment.

css-meeting-bot commented 7 hours ago

The ARIA-AT Community Group just discussed Bot Run Assignment Failed.

The full IRC log of that discussion <jugglinmike> Topic: Bot Run Assignment Failed
<jugglinmike> github: https://github.com/w3c/aria-at-app/issues/1226
<jugglinmike> IsaDC: The original issue was about the bots not getting assigned to the tests--any of the bots, neither VoiceOver Bot nor NVDA Bot
<jugglinmike> IsaDC: It worked in the sandbox server, and the VoiceOver Bot improved a lot--that's great!
<jugglinmike> IsaDC: Today, I tried again, and the bots received the assignments, but the test run is cancelled
<jugglinmike> IsaDC: They do not report a recording
<jugglinmike> howard-e: I submitted a patch for this yesterday, but it hasn't been merged, yet
<jugglinmike> howard-e: There was a regression which was causing the bot to attempt to run the test with the earliest AT version possible
<jugglinmike> howard-e: That was an AT version that the automation driver doesn't have
<jugglinmike> howard-e: The patch should address this
<jugglinmike> howard-e: A quick fix (which I do not recommend) would have been to create a test plan specifically for automation
<jugglinmike> IsaDC: I tried to do that, but it didn't work
<jugglinmike> howard-e: I think it's NVDA 2023.3 and VoiceOver version 14
<jugglinmike> IsaDC: I believe that's what I tried
<jugglinmike> howard-e: I will have to verify, then
<jugglinmike> howard-e: The patch is available on staging. I'll confirm again there, then send a message, and finally push to production
<jugglinmike> Matt_King: Is that regression something that you think we need a test for?
<jugglinmike> howard-e: There is a test on its way
<jugglinmike> howard-e: There was another critical issue due to this same regression. It was reported by Dean on the mailing list yesterday. There's also a patch coming for that
<jugglinmike> howard-e: Dean reported that they were unable to submit the tests that they've completed for the "navigation menu button". I verified that. The issue is with how assertions are excluded by the navigation menu button.
<jugglinmike> howard-e: The regression occurred when we made a change back in August, but we haven't seen it in any test plan we've worked on since that time
<jugglinmike> howard-e: I'm submitting a patch (a patch to fix this) for review right now