"Activate element" user intent

[jugglinmike]

The "pressKeys" interaction currently defined in AT Driver enables the simulation of HID-level keyboard events. However, because such a capability undermines the security model of some platforms, it isn't a generally-viable solution.

In recognition of this state of affairs, we anticipate addressing most automation needs without relying on keyboard simulation. We intend to do this by specifying a collection of purpose-built "user intents" that model user interactions at a higher level (e.g. "move to next heading"). Although such a collection could be expanded to allow for the automation of most kinds of interaction with a given assistive technology, it would not be able to address one critical scenario: the filling of form fields.

In order to allow clients to interact with form fields without a means to simulate HID-level keyboard events, we envision a new user intent which causes the focus of the AT's "virtual cursor" to become the active element of the document to which it belongs. Clients wishing to fill form fields could use an API like that in conjunction with WebDriver's "element send keys" command (or the equivalent in WebDriver BiDi) to accomplish their goal.

(For more context on this design, see gh-79 and the discussions referenced there.)

Add a user intent that instructs the screen reader to induce the browser to make the virtual cursor the "activeElement" of its containing document. Explore ways to discuss the intended use-case via additional non-normative text.

[jugglinmike]

Here's a quick rundown of the relevant functionality in a few implementations:

AT	feature	Desktop key	Laptop key
NVDA	Move System focus or caret to current review position	NVDA+shift+numpadMinus	NVDA+shift+backspace
JAWS	Route JAWS Cursor to PC Cursor	INSERT+NUM PAD PLUS	CAPS LOCK+APOSTROPHE
VoiceOver	n/a (document focus is expected to follow virtual cursor)	n/a	n/a

Notably, NVDA also supports a motion it calls "Activate current navigator object." That motion certainly sounds more similar to the "user intent" proposed by this issue. However, despite the apparent similarity, I believe "activate current navigator object" is actually a different semantic. The description reads as follows:

Activates the current navigator object (similar to clicking with the mouse or pressing space when it has the system focus)

My understanding is that the "activate element" user intent in AT Driver ought to only give DOM focus to the element under the virtual cursor (thus updating the value of document.activeElement) without interacting with the element any further.

[css-meeting-bot]

The ARIA-AT Community Group just discussed "Activate element" user intent.

The full IRC log of that discussion

<jugglinmike> topic: "Activate element" user intent
<jugglinmike> github: https://github.com/w3c/at-driver/issues/81
<jugglinmike> jugglinmike [asks clarifying question]
<jugglinmike> s/jugglinmike /jugglinmike: /
<jugglinmike> Matt_King: if the virtual cursor is already on an element then if you press enter, it activates the element. that is more equivalent to doing a mouse click. that's essentially what it does (simulating a mouse click)
<jugglinmike> jugglinmike: this was prompted because i misunderstood what we wanted by activate elemtn. i thought we wanted something where we'd update the document focus to be the element where the virtual cursor goes
<jugglinmike> Matt_King: yes and that's exactly what Enter does. no activate element could also mean if you are on a checkbox, it checks it, on a radio button then it pushes
<jugglinmike> jugglinmike: okay so i'm now learning that's what we want
<jugglinmike> Matt_King: yes, i'm pretty sure this is the intent we want for activate element. now interestingly, there are scnearios where ctrl+opt+space. maybe by default it moves focus but you have to change a setting to make it also move focus. so we can define that as the intent. Both set element.activeelement to that element and simulate a mouse click
<jugglinmike> jugglinmike: okay, totally fine. we can specify what we want. Let's just make sure we're specifying the right thing here. I previously wasn't even considering VO. But actually if the semantics is more broad than that (activating element and do mouse click) then it would be a meaningful command
<jugglinmike> Matt_King: Okay great
<jugglinmike> jugglinmike: Thank you howard-e for minuting that
<jugglinmike> Zakim, end the meeting