Improve Checker capabilities with Chatbot-like implementations that refresh the page constantly with end-user input

[philljenkins]

Pages with Chatbot implementations often delay the loading and/or frequently update the page. The Checker users, especially those who are not developers of the page being tested, do not know that the Chactbot contents (or page) were changed, or will soon be changed, which results in confusing Checker results, especially in the Keyboard Checker Mode visualizations.

Some ideas and approaches to consider:

1. Notify the user that the page has changed since the last scan. Also see #1056
2. Notify the user that the page has changed since the KCM visualization was drawn and may therefore be inaccurate
3. Prompt the user to re-scan that page and re-run KCM if it has currently been run
4. Add a setting (Option) to clear the results and clear the KCM when the page has changed
5. Add a setting (Option) to refresh the results (re-scan automatically) and re-draw the KCM when the page has changed. every xx minutes

Test with these IBM ChatBots:

Use these 3 chatbot implementations as test cases to determine how best to improve the Checker for checking AI Chatbots:

• AskHR
• AskIT
• AskAccessHub Assistant Security, control and ease of use for IBMers

Address the following questions and explore best approaches:

• How should the Checker handle known-to-be delayed loading? For example, AskHR digital assistant loads the page, places the input cursor into the chat input field, then very quickly adds an additional announcement to the conversation, then moves the keyboard focus to a link in the announcement, and then finally pauses until the end-user types in a query. Note that the keyboard focus is never placed back into the chat input field.
• How should the Checker handle known-to-be-often apps that change the page such as a conversation-like chatbot app? Can timers be added that are triggered to check and then notify the user when the page changes? Such a timer would need to be app-like specific, even Chatbot-specific.
• How should the Checker handle check if the Chatbot has NOT updated the page since the last scan? for example, the page hasn't changed, so no need to re-scan. A type of exception to the setting of asking the Checker to check (often, every 5 mins, etc.) if the page needs to be re-scanned.

How to reproduce the issue

• [ ] Open Checker on a Chatbot, such as AskHR, scan, and run Keyboard Checker Mode
• [ ] Carefully note page & content changes before and after the scan, including keyboard input cursor changes
• [ ] Identify possible additional approaches to those 5 listed above

What to do

• [ ] Review with Design, however, this is more of a technical discussion on what the Checker can do.
• [ ] Prioritize which scenarios are in scope to make improvements with Checker

Why is this important?

• AI Chatbots are strategic, and accessibility is critical, so Checker needs to make the task of checking for accessibility easier, more efficient, and more effective
• US Government and the Whitehouse has issued an executive order (Nov 2023) to require accessibility to [people with disabilities as part of its AI Trustworthy mandate.

[philljenkins]

Phill and Josiah met briefly and determined to open an issue to determine future consideration and prioritize our recommendations.

[philljenkins]

Checking the AskHR Chatbot shows the complexity, in this screenshot, the component in error is off-screen

• the keyboard focus was automatically moved by the developer from the chat input to a notice in history
• the conversation content is constantly changing with user input, how should the Checker respond?

[shunguoy]

triage: rewrite the issue,

• [x] Done by Phill

also see #1056 about how the Checker KCM should handle components that float on the page.

[philljenkins]

@marcjohlic Issue re-written to create a prioritized list of possible improvements to the Checker, as it relates to Chatbots. Then, to-be-determined if one or more can then be implemented in this release.

[philljenkins]

@tombrunet Do you want to make these/one of these approaches more generic or as proposed more specific to common Chatbot-like behaviors?

[philljenkins]

Moving to Release backlog to be considered in this release.

[philljenkins]

This is not an Engine Issue, but an Extension issue (changed Epic). No rule changes are needed or are in scope. Because this is about notifying the user and/or adding a setting, Design should be included.

However, this initial task is more of a technical discussion (as noted above in the description) to determine what the Checker can do.

[philljenkins]

@shunguo added:

an auto-page refresh has other impacts too. For example, go to https://finance.yahoo.com/calendar/, scan the page, wait a little while, then click a Violation (message row), user will see the original scan results disappear (depending on timing) and a message replacing it: "This page has not been scanned." displayed. Checker extension needs a better handling of the developer-side (not user initiated) page refresh.

But this (reload whole page vs change in DOM) is probably a separate issue/feature. The user is told by the Checker that the page has not been scanned, although not told it was "re-loaded" and not told why. Also, the Checker does not tell that this is a separate accessibility issue that the developer should fix, namely reloading (WCAG uses "auto-updating") the page content without end-user knowing about it:

2.2.2 Pause, Stop, Hide (Level A) For moving, blinking, scrolling, or auto-updating information, all of the following are true: ... For any auto-updating information that ... there is a mechanism for the user to pause, stop, or hide it or to control the frequency ... unless ... essential.

3.2.5 Change on Request (Level AAA) Changes of context are initiated only by user request or a mechanism is available to turn off such changes.