Alt-Input and Support for 1D vs 2D dragging

In Slack#a11y-dev from 10/4/2022 to 10/11/2022:

Chris Malley 7 days ago In pH Scale, I have some objects that are constrained to “drag” only in 1 dimension — up/down or left/right. The KeyboardDragListeners for those objects enforce the same constraints. That is, if my object can be dragged up/down, then the left/right arrow keys do nothing. But I see that all of the common-code keyboard help shows the full set of arrow/WASD key icons, with no option to exclude those that do nothing. Is the expectation that (for example) left/right arrows should still do something for an object that is constrained to up/down dragging? Or should the keyboard help only show the arrow/WASD keys that are relevant? (edited)

Chris Malley 7 days ago I’d like to hear from @terracoda

@jessegreenberg

@mjkauzmann

@arouinfar . Thanks.

Jesse Greenberg 7 days ago 1D motion sounds like a good case for AccessibleSlider instead of KeyboardDragListener.

Chris Malley 7 days ago Oh no…. Nothing about that in the alt-input-quickstart, and I spent an entire day adding KeyboardDragListeners. (edited)

Chris Malley 7 days ago KeyboardDragListener was also very straightforward, because it reused the same code as the existing DragListener. Will that also be the case with AccessibleSlider?

Jesse Greenberg 7 days ago Its probably fine (especially for just alt input), I see it working well in pH scale.

Jesse Greenberg 7 days ago No, it probably won't share as much with DragListener

Chris Malley 7 days ago If it’s not going to share with DragListener, then I don’t think we should use AccessibleSlider here. By “working well in pH Scale”, I’m assuming that means it’s OK to ignore the arrow keys that are not relevant — do I understand correctly? And should keyboard help only show the arrow keys that are relevant? (edited)

Chris Malley 7 days ago Specifically, can I show this for the Graph Indicators in the My Solutions screen:

Chris Malley 7 days ago Or should I remove the left/right/A/D key icons? Graph Indicators are constrained to vertical dragging.

Chris Malley 7 days ago These are the Graph Indicators:

Jesse Greenberg 7 days ago By "working well" I meant using KeyboardDragListener seems good to me. I thought you hadn't gotten there yet. I would recommend (and probably easiest thing) would be to include those keys. Otherwise we can modify keyboard help.

Jesse Greenberg 7 days ago But it would be good to hear from design.

Chris Malley 7 days ago Do you mean include those keys in the keyboard help, or also make them do something?

Jesse Greenberg 7 days ago I meant make them do something. up/right do the same and down/left do the same.

Chris Malley 7 days ago I see for example in GFL that the “up” arrow moves the mass to the left.

Chris Malley 7 days ago Sorry… “up” moves to the right in GFL. OK, I’ll do that.

Jesse Greenberg 7 days ago Yep, thats an example of AccessibleSlider though.

Chris Malley 7 days ago From the user’s viewpoint, why shouldn’t they behave the same?

Jesse Greenberg 7 days ago Sounds good! Just noting in case you went looking there for an example.

Chris Malley 7 days ago Ugh… Making “up” move to the right with KeyboardDragListener is not at all straightforward. For example, my current implementation in PHDropperNode is:

   this.addInputListener( new KeyboardDragListener( {
      dragVelocity: 300, // velocity of the Node being dragged, in view coordinates per second
      shiftDragVelocity: 20, // velocity with the Shift key pressed, typically slower than dragVelocity
      positionProperty: dropper.positionProperty,
      dragBoundsProperty: new Property( dropper.dragBounds ),
      transform: modelViewTransform,

… and KeyboardDragListener takes care of changing postionProperty, apply transform, respecting drag bound, etc. (edited)

Chris Malley 7 days ago Am I now going to have to handle all of that myself, based on which keys are pressed?

Jesse Greenberg 7 days ago Hmm, yes I guess so, I don't see an easy way to remap keys

Jesse Greenberg 7 days ago Should we add an option to override default directional movement keys? So you could say rightMovementKeys: [ KeyboardUtils.KEY_RIGHT_ARROW, KeyboardUtils.KEY_D, KeyboardUtils.KEY_W, KeyboardUtils.KEY_UP_ARROW ]

Jesse Greenberg 7 days ago I don't think you should need to handle the positioning yourself for this case.

Chris Malley 7 days ago That would be helpful, if KeyboardDragListener is appropriate for 1D movement. Perhaps there are other reasons why you recommended AccessibleSlider. But a big reason NOT to use AccessibleSlider is that it makes it more difficult to add alt input, because it’s not as easy for the implementation to reuse the existing DragListener.

Chris Malley 7 days ago AccessibleSlider would also seem to be a bit of an odd choice semantically. As a user, I would consider the eye dropper (for example) to be “one of the things I can drag”, not a “a slider”. And that UX issue flows through to keyboard help. (edited)

Chris Malley 7 days ago And unlike a slider, dragging the eye dropper left/right doesn’t control any value that’s meaningful to the sim or user. It just moves the dropper to a different position. (edited)

Jesse Greenberg 7 days ago My first recommendation for AccessibleSlider was because 1) it is good for 1D motion, 2) AccessibleSlider is much easier for description (and better for the screen reader user) when we get there.

Jesse Greenberg 7 days ago dragging the eye dropper left/right doesn’t control any value that’s meaningful to the sim or user. True, AccessibleSlider controls a Property so a Property like this would have to be created just for this. Also, ACcessibleSlider will move the object in discrete steps, which may or may not be better.

Chris Malley 7 days ago Looks like I could override the following methods in KeyboardDragListener. Is that appropriate or recommended?

leftMovementKeysDown
rightMovementKeysDown
upMovementKeysDown
downMovementKeysDown

Jesse Greenberg 7 days ago Oh, that could work! Do you think that is better or would you rather have an option for each set of direction keys?

Chris Malley 7 days ago Options would be better, so that all 1D KeyboardDragListeners don’t need to create a subclass and override those methods. (edited)

Jesse Greenberg 7 days ago OK! https://github.com/phetsims/scenery/issues/1460

Chris Malley 7 days ago It might even be better to have:

type KeyboardDragDirection: 'leftRight' | 'upDown' | 'both';

// KeyboardDragListenerOptions
keyboardDragDirection?: KeyboardDragDirection

// default option value
keyboardDragDirection: 'both'

… and then standardize how those behave. And option like rightMovementKeys would not be standardized.

Jesse Greenberg 7 days ago That would also be easier to implement.

Chris Malley 7 days ago Bonus!

Chris Malley 7 days ago I tried the approach of overriding KeyboardDragListener methods and it works. See GraphIndicatorKeyboardDragListener. In this case, it’s not terrible, because I’m already subclassing KeyboardDragListener. But it’s not great, because it relies on me having specified the correct combinations of keys for up/down. :+1: 1

Taliesin Smith 7 days ago Isn’t KeyboardDragListener for sliders?

Jesse Greenberg 7 days ago It is not unfortunately. KeyboardDragListener is used for custom dragging. AccessibleSlider uses the input events that we get from a browser and AT specifically for interaction with <input type="range"> elements.

Chris Malley 7 days ago KeyboardDragListener is for general dragging, the keyboard equivalent of DragListener for mouse/touch. Documentation in KeyboardDragListener.ts:

/**
 * A general type for keyboard dragging. Objects can be dragged in two dimensions with the arrow keys and with the WASD
 * keys. This can be added to a node through addInputListener for accessibility, which is mixed into Nodes with
 * the ParallelDOM trait.
 *

Emily Moore 7 days ago Not sure if this has been brought up, but my understanding is that slider interactions are very much preferable to implementing custom interactions for many reasons, including usability. Has that been brought into the discussion?

Chris Malley 7 days ago Understood. But this isn’t really a custom interaction. There are well-defined and standardized interactions for things that can be dragged via a KeyboardDragListener, and the a11y team has created standardized keyboard help for these things (arrow keys, WASD keys). The question is which standard behavior is appropriate for things that are constrained to drag in 1 dimension. Not all of these things look/behave like sliders, so AccessibleSlider may not be the best fit. And using a KeyboardDragListener for 1D/2D dragging is much easier to implement/test/maintain in cases when a DragListener already exists for mouse/touch interaction, because the DragListener and KeyboardDragListener can share most of their behavior and code. That said, I’m looking to the a11y team for guidance on how to proceed. (edited)

Chris Malley 7 days ago Regardless of which way we go for pH Scale, I think the additional KeyboardDragListener options that @jessegreenberg proposed are an important/useful addition.

Jesse Greenberg 7 days ago I described some reasons to use AccessibleSlider in https://phetsims.slack.com/archives/CA04ERJJK/p1664916006808179?thread_ts=1664913917.449129&cid=CA04ERJJK and the following comment. For alternative input I actually prefer the feel of KeyboardDragListener, especially with the code considerations. But either could be used. We should describe AccessibleSlider in the alt input-quick start guide.

Jesse Greenberg My first recommendation for AccessibleSlider was because 1) it is good for 1D motion, 2) AccessibleSlider is much easier for description (and better for the screen reader user) when we get there. From a thread in a11y-dev | Oct 4th | View reply

Chris Malley 7 days ago

My first recommendation for AccessibleSlider was because 1) it is good for 1D motion,

It seems odd to me that a draggable thing suddenly changes from a KeyboardDragListener to an AccessibleSlider when its dragging is constrained to 1D. It also has implications for PhET-iO, where the client will be presented with any entirely different Studio tree structure depending on whether something can be dragged in 1D and 2D. Given the amount of time the PhET-iO team has been putting into providing a consistent structure/experience for the Studio tree, perhaps that’s something to consider. @kathy

@samreid

@mjkauzmann

2) AccessibleSlider is much easier for description (and better for the screen reader user) when we get there.

What aspect of description are you referring to? According to https://github.com/phetsims/phet-info/blob/master/doc/interactive-description-technical-guide.md#what-does-interactive-description-mean, alternative input is description. Do you mean State Description, Response Description, or both? Why is AccessibleSlider “easier” for this — is it inherent in AccessibleSlider design, or just that the implementation is more mature? (edited)

Chris Malley 7 days ago @jessegreenberg said above:

Also, ACcessibleSlider will move the object in discrete steps, which may or may not be better.

For pH Scale, moving the Graph Indicators (which are constrained to move up/down) in discrete steps would be a problem.

For alternative input I actually prefer the feel of KeyboardDragListener, especially with the code considerations.

Yes, I agree. If you want to play with pH Scale, I’ve implemented the “override KeyboardDragListener” workaround for the dropper and Graph Indicators. With the new KeyboardDragListener options proposed in https://github.com/phetsims/scenery/issues/1460, I think this will be a nice API and UX. (edited)

Chris Malley 7 days ago So back to the “easier for description” aspect, so that we’re future-proofing… What can we do to make that easy for KeyboardDragListener, or at least on par with AccessibleSlider? (edited)

Taliesin Smith 7 days ago That’s a good question, and I love this deep conversation. Input type range has a native method for providing built-in object responses known as aria-valuetext. This usually makes for a very good screen reader experience, but it can depend on the interaction. In RaP we have the Indy slider hands and then a custom 2-way Both Hands interaction for bi-manual input. Also, regarding screen reader access, while we have some nice custom interactions that work great with alt input, when using screen reader these interactions can come with description baggage that we have no control over. For example, VoiceOver says, “Both Hands, empty movable” rather than just “Both Hands, movable”. Native interactions are and sound more familiar, for example “At challenge ratio, Left Hand, slider” or in GFLB, “2.5 kilometres from Mass 2, Move Mass 1, slider”

Taliesin Smith 7 days ago But I think we should continue this conversation because if keyboard drag listener handles dragging on touch this would be good to make more inclusive. Currently, our sliders on mobile only work with swipes (discrete input) but mobile VoiceOver users can double tap and slide their finger on the screen…they just won’t get description. That would be a nice issue to solve. (edited)

Michael Kauzmann :house_with_garden: 6 days ago Hey all! I'm joining the conversation late here. @pixelzoom said:

So back to the “easier for description” aspect, so that we’re future-proofing… What can we do to make that easy for KeyboardDragListener, or at least on par with AccessibleSlider? (edited)

Inherently this interaction will be less intuitive, because it doesn't directly map to HTML that a user would be used to be using (i.e. form controls). It will most always be an "application" role on items with a keyboard drag listener, which is basically saying to the user "this is different, good luck". Then we have tools to help the user learn what to do. When we map to a slider, it greatly reduces friction for the user, which will likely always be the fist choice for me when deciding how to implement. This is why we use keyboardDragListener as sparingly as possible, even when it would be much, much easier for alt input to have it. :+1: 1

Chris Malley 6 days ago

This is why we use keyboardDragListener as sparingly as possible, even when it would be much, much easier for alt input to have it.

This is the first that I’ve heard of this. The alt input quickstart guide instructs us to add a KeyboardDragListener wherever there is a DragListener. I understand how AccessibleSlider can fill this role for 1D dragging. But what is the alternative when dragging is 2D?

I also do not see KeyboardDragListener used “sparingly”. It’s used in many (most?) of sims that support alt input: area-model-common (4 sims), balloons-and-static-electricity, coulombs-law, fourier-making-waves, friction, geometric-optics, gravity-force-lab, ph-scale, and quadrilateral. And any sim that uses GrabDragInteraction, which requires a KeyboardDragListener. I’m still very unclear on how (in general) I should be adding alt input support for Nodes that have a DragListener. (edited)

Taliesin Smith 6 days ago I think MK's statement might be referring more towards when we are incorporating description for screen reader access.

Taliesin Smith 6 days ago Screen readers in general do not handle custom objects very well (i.e. things that cannot be coded as native HTML form elements). These are very good things you are bringing up.

Kathy Perkins 5 days ago Following this good discussion. Definitely a lot of complexity around choices. My understanding is what Taliesin said - for description, slider is the current go-to choice for dragging things. Balloons and Static Electricity is the sim that has 2d drag I think. The work on pH Scale is in the context of scaling the suite of inclusive features - alt-input, pan-zoom, interactive highlights, and UI Sound. These are things we can scale at the moment, but working on pre-structuring these sims for description is beyond the scope of what we can do - too much time and too costly to restructure code in these ways. We wouldn't be able to take on scaling these features if we overload the goal with pre-design for description. There is a lot of work and fundraising ahead of us to scale description to more sims. For scaling alt-input to more sims, using keyboardDragListener has been the straightforward approach (efficient, user-friendly, and does not require restructuring of code) for sims with dragListener that are working on the suite of features we can currently support at scale alt-input, pan-zoom, interactive highlights, and UI Sound. As we will discuss today, we need to spend time on infrastructure innovation and building before we have a viable path to bring interactive description to scale. This work may open up more options around how to address this challenge around dragListener and description. (Or it may not).

Chris Malley 5 days ago :+1::skin-tone-2:

Taliesin Smith 5 days ago I'd second @kathy ’s comment that you don't have to use a slider under the hood for every 2-d drag interaction. The puller faucets in pH Scale actually don't appear like or behave like slider interactions to me. They might be something different - a special kind of button. It's nice (when helpful) to look to the future to see if one can choose a native HTML element for the PDOM that might be a good fit for description. It could save time in the future, but it is not required to do that and the web and screen readers will change. When we are ready to do description many things can change. We can cross that bridge when we come to it.

Emily Moore 3 hours ago I’m following along here and want to note - it is highly unlikely from my understanding that we would not want to change the vast majority of 1-D sliding behavior into a an Accessible Slider. I worry that this is not the right spot to draw a line indicating “we’ll deal with this later when we’re scaling interactive description”. If we know that we would want to outfit those interactions differently for interactive description, I would suggest that we proceed that way now. There are significant benefits re non-visual access to having anything that can be “slider-like” instrumented as an Accessible Slider, and we have proceeded in doing so for all sims my team has worked on. I would recommend proceeding in that way, unless @terracoda

@jessegreenberg

@mjkauzmann see something else in this discussion that would lead you to recommend otherwise.

Emily Moore 3 hours ago @terracoda

for pH Scale, I think the sliding nobs/readouts associated with the pH scales have 1-D behavior and map nicely (conceptually) to Accessible Slider behavior. I agree the faucet nobs are more like buttons.

Kathy Perkins 2 hours ago From my perspective, its about implementation hours and cost at the moment. The more re-work/pre-work we load into a sim to add alt-input right now, the fewer sims we can afford to add alt-input into at this time because it will add too much extra implementation time for the schedule. So in my mind it comes down to a trade off of implementation approach and number of sims with alt-input. I would like to choose more sims with this feature set. If we secure some explicit funding to support this scaling, that changes the parameters here. (Note - here I am talking about propagating alt-input back into existing code) (edited)

Taliesin Smith 2 hours ago I would like to frame the question differently. What is making it difficult for the broader team to recognize interactions that already fit well into a given PhET component that has already been out-fitted with an accessible design pattern? Or what is it about the accessible design patterns that is making it difficult for them to be scaled up and be used/understood by the broader team?

Taliesin Smith 2 hours ago I have design pattern documentation issues assigned to me. Will they be used for implementation of alternative input?

Kathy Perkins 1 hour ago Hey Taliesin, I think there are two parts here ... articulating best practices for design approaches for new sims that match with the accessible design patterns (this I think this is right in line with what your are focused on here and a goal we listed for this quarter that Amanda suggested a few weeks ago around documenting the current design/development process these days). The second piece - and the piece that my comment above was around - is regarding bringing a subset of features to existing sims at this time given the code and code structure that has already been written.

Chris Malley 1 hour ago @terracoda asked:

What is making it difficult for the broader team to recognize interactions that already fit well into a given PhET component that has already been out-fitted with an accessible design pattern?

In general… There is very little documentation of design for specific UI components, and no documentation (that I’m aware of) for general accessible design patterns. Even in the code, there’s very sparse documentation about the what’s/why’s of code related to a11y. It’s unclear (from the code) whether what is implemented was actually designed, or whether it was a quick prototype to address the needs of a specific sim. And there has been a pattern of not involving the original authors of UI components in the design/implementation of their a11y features.

FaucetNode as an example… I’m the primary author of that UI component, and I was present at for all of the design that occurred prior to adding alt input support. I was never consulted about a11y features, or consulted about the a11y design. The a11y features that were added are incomplete — they don’t address all of the interactions that are possible (needed?) with FacuetNode. There is no documentation related to how FacuetNode uses accessible design patterns, either in the code or externally (.md file). And there has been input from others (see eariler in this thread) that perhaps FaucetNode’s current alt input behavior needs another look.

Or what is it about the accessible design patterns that is making it difficult for them to be scaled up and be used/understood by the broader team?

The lack of documentation is one thing that I can point to. There are very few .md files that describe components that have been outfitted with accessibility. In the case for FaucetNode, there is no documentation. So I typically can’t tell if there was an accessible design process, whether it was intended to be a complete solution, whether the implementation that I’m looking at was a prototype to quickly address the needs of a specific sim.

I have design pattern documentation issues assigned to me. Will they be used for implementation of alternative input?

It’s more likely that design and implementation would be iterative. And ideally the design documentation should be kept in sync with the implementation. If it becomes stale, then its usefulness is diminished, and it can even become confusing. (edited)

Sam Reid :face_with_thermometer: 1 hour ago This thread looks like it has lots of valuable information, can we archive it in an GitHub issue for future reference and review? :+1::skin-tone-2::+1: 4

New

Taliesin Smith 13 minutes ago Please do so!

Chris Malley 3 minutes ago I realize that I started this thread. But I’m on vacation, so it would be great if someone could archive.

phetsims / scenery

Alt-Input and Support for 1D vs 2D dragging #1475