Develop a design pattern for Alt input of toolboxes/creator pattern

zepumph commented 2 years ago

This issue has been in many people's minds for quite some time, but most recently this has come about in https://github.com/phetsims/geometric-optics/issues/258.

Please make sure to go look at that GO issue, as there was quite a bit of discussion about the general design over there.

This is likely scheduled for work during Q1 2022. We will proceed from there. It is currently blocking the publication Geometric Optics, though that could potentially change.

I was curious about how challenging it may be to implement a basic usage of this, in which the toolbox icons are buttons, and when clicked, they focus the draggable Node. In waves-intro, I was able to add an example of this quite easily! It took ~30 minutes to get something that feels solid enough to inform a design meeting for it. I plan to commit these changes to a branch, and publish a one-off version of waves-intro for use in the design process.

zepumph commented 2 years ago

I created a prototype that is a bit rough, but could be used in the design process. See it implemented for the stopwatch node on the first screen of https://phet-dev.colorado.edu/html/waves-intro/1.2.0-altInputToolbox.1/phet/waves-intro_all_phet.html

Current implementation:

icon is a button
clicking it shows, centers, and focuses the draggable node
draggable node is a focusable 'div'
drag the element around with KeyboardDragListener
drag it back to the toolbox to hide it

Still to do/some random questions I had:

use escape key to put the toolbox back
make button look more clickable somehow? Like a focus state?
keyboard shortcuts dialog
dashed focus highlight around the draggable node once pulled out

marlitas commented 7 months ago

Meeting 3/6/24:

A foundational alt-input pattern was implemented here, but we do agree that more work can be done to improve the user experience as well as compatibility with description.
@terracoda will document the current alt-input pattern in this issue then assign the issue back to @zepumph and @jessegreenberg to include in the alternative-input-quickstart-guide.
A side issue about grouping toolbox items for an improved UX should also be created.

terracoda commented 6 months ago

The design pattern for navigating and grabbing tools from a toolbox: Using Geometric Optics as an example.

Tab to toolbox item (no grouping at this point). Each tool is essentially a "grab" button.
Tab or Shift Tab to the next tool or previous tool in the tollbox
Activate or grab the tool with Space or Enter.
Move the draggable tool with Arrow keys or W,A, S, or D keys.activated tool with arrow keys.
Return the tool to the toolbox with Escape.
Optional: Design sim-specific shortcuts to jump the tool to useful places.

Screenshot 2024-03-11 at 16 33 13

terracoda commented 6 months ago

What's missing in GO and GO:B is any visual hint that one needs to grab a tool with Space.

terracoda commented 6 months ago

I think for the toolbox/creator pattern https://phetsims.github.io/scenery/examples/creator-pattern.html, the two-step Grab and Drag designed for GO and GO:B is really good and works well

I do not think a second two-step grab interaction once the tool is activated and in the Play Area is necessary.

Important things to ensure/consider:

The PDOM placement of the activated tool makes sense to the use of the activated tool.
Focus goes on the activated tool so it can be easily moved once activated.
It's not too easy to accidentally return the tool to the tool box when using Alt Input.
The Escape key works to return a focused-activated tool to the tool box.
Would any sim-specific shortcuts make using the tool more efficient and effective for Alt Input users.

Workarounds and other considerations:

The checkbox interaction can be used as a work-around, but the Escape won't work as a shortcut in this case to hide the tool and uncheck the checkbox.
A tool activated by a checkbox also does not need an additional two-step drag. Like in the case of the toolbox, the user intentionally activates the tool, and the arrow keys are quite intuitive for draggable objects.
Tools activated by a checkbox would also need to consider the need for sim-specific shortcuts.

Description Design Consideration for the Future

The tool activation step for both the toolbox and the checkbox tool provides the opportunity to give instructions to blind users once description is implemented. The activation step is what was determined to be needed for BASE's Yellow Balloon, Friction's Chemistry Book, and others (FL, GFLB) where these draggable objects are already in the Play Area.
It's these "already-in-the-Play-Area" draggable sim objects that are what is controversial and that may need to be revisited once the sims that contain these readily available draggable objects get a description feature.

To Do's for Toolbox/Creator pattern (future work)

Create a re-usable visual hint for grabbing a focused tool from the toolbox.
The hint would need to be platform dependent and there are open issues related to this (like https://github.com/phetsims/scenery-phet/issues/704).
In the KB Help dialog, consider having the instructions for activating the toolbox/creator objects before the instructions for dragging in the objects. This is not the case in GO and GO:B.
Consider grouping all icons in a toolbox in one focusable entity, using arrows to cycle through the "creator buttons", (like https://github.com/phetsims/sun/issues/874 and https://github.com/phetsims/scenery-phet/issues/844)

terracoda commented 6 months ago

@zepumph, I think I documented the design pattern for the toolbox/creator pattern, and I included design considerations and to-dos. I am not sure this all constitutes what we need.

@marlitas 's comment in https://github.com/phetsims/a11y-research/issues/166#issuecomment-1981577046 seems to be about "already-in-the-Play-Area draggable sim objects" which seem different than draggable tools activated from a toolbox or by a checkbox.

Should I also include a design pattern that documents the current alt input approach (i.e., no two-step drag) for readily available draggable sim objects - sim objects that are not activated from a toolbox or by a checkbox?