search

phetsims / balloons-and-static-electricity

"Balloons and Static Electricity" is an educational simulation in HTML5, by PhET Interactive Simulations.
http://phet.colorado.edu/en/simulation/balloons-and-static-electricity
GNU General Public License v3.0
6 stars 10 forks source link

New balloon interaction: 'Play with balloon' button #133

Closed jessegreenberg closed 8 years ago

jessegreenberg commented 8 years ago

In #116, @terracoda suggested a new balloon interaction pattern:

One other idea is to change the interaction flow by adding an explicit grab before the drag. I think this was my original idea way back when. A comment from one of the participants supports this interaction pattern. He mentioned that in gaming, he often uses the Spacebar to pickup AND drop things.

With an example interaction pattern:

user moves focus to balloon with the Tab key and hears "Play with Balloon, button". user presses Spacebar (or Enter) to play with the Balloon. Nav/interaction cue is read out. "Use W, A, S...to drag balloon up, left.... Press Spacebar to release."

jessegreenberg commented 8 years ago

Three possibilities:

  1. Invisible buttons exist in the PDom that sets focus to balloon when selected. Button has its own labels and descriptions to match pattern above. Button should be hidden when balloon is being played with, and hidden when balloon is invisible.
  2. BalloonNode dynamically resets its accessible peer from button to application div as necessary
  3. BalloonNode remains a div, toggling between `role=applicationandrole=button`` as necessary given current point in interaction.

At this point option 1 seems best. I want to avoid option 2 if possible, restructuring DOM is complicated and could be disorienting to the user. Option 3 would be best, but role=button does not work very well across platforms. See http://unobfuscated.blogspot.com/2013/05/event-handlers-and-screen-readers.html. That article is three years old, but I have observed similar performance.

jessegreenberg commented 8 years ago

As I work on this I notice that this could be confusing to those that do not use a screen reader. They will need some direct cue that a 'pick up' interaction is required.

jessegreenberg commented 8 years ago

Option 1 is not possible. In order to correctly maintain the focus highlight around the balloon for all steps in the interaction, the accessible peer and content needs to change for the node. Now I think setting balloonNode.accessibleContent (option 2) is the best approach. In https://github.com/phetsims/balloons-and-static-electricity/issues/121 we determined that a div with role=button does not behave well.

jessegreenberg commented 8 years ago

Option 2 is working very well with NVDA + Firefox (including arrow keys!). In chrome, no keys work with NVDA and the 'application' content for the balloon is not read out unless the user navigates through those descriptions with the virtual cursor.

terracoda commented 8 years ago

@jessegreenberg The unobfuscated post also advises against using a div with the role button. Another helpful resource might be Léonie Watson's talk on Accessibility Mechanics. APIs and ARIA. Her slides are posted on slide share if you need a closer look. I envisioned the grab interaction as a control separate from the balloon, but didn't think about how this would affect mouse users. Perhaps, it makes sense to have a setting to turn on keyboard interactions or turn a screen reader interactions. If the user tells us upfront they are using a screen reader or the keyboard we can tell them upfront how to use the sim. The plot thickens :-) Sorry, I have to write elsewhere. I hope to be back here soon.

jessegreenberg commented 8 years ago

Very cool, thanks for the resource @terracoda. I will take a look at

The way it is implemented now should have no impact on the interaction for mouse users. Here is the way it is implemented with https://github.com/phetsims/balloons-and-static-electricity/commit/f748bc2e38d12f07ab40d9ce38b1cae436a4d6f1.

<button>Play with ‪Yellow Balloon‬</button>
<div tabindex="0" role="application" id="balloon-17-42-45-278-351" draggable="true" aria-grabbed="true" class="Balloon" aria-labelledby="balloon-label-17-42-45-278-351" aria-describedby="balloon-description-17-42-45-278-351 navigation-description-17-42-45-2i 78-351">
    <h3 id="balloon-label-17-42-45-278-351">‪Yellow Balloon‬</h3>
    <p aria-live="assertive" id="balloon-description-17-42-45-278-351">‪Yellow Balloon has a ‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪net negative‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬ charge, ‪several‬ more negative charges than positive ones.‬</p>
    <p id="navigation-description-17-42-45-278-351">‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪Use W A S D keys to grab and drag balloon up, left, down and right. Tab for next object. Press question mark for keyboard commands and help.‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬</p>
</div>

This way the focus highlight will remain on the balloon for each step in the interaction.

Perhaps, it makes sense to have a setting to turn on keyboard interactions or turn a screen reader interactions. If the user tells us upfront they are using a screen reader or the keyboard we can tell them upfront how to use the sim.

Interesting possibilities! This would align with recent brainstorming concerning layers and settings for accessible content.

jessegreenberg commented 8 years ago

I would like to test with JAWS and have this reviewed in a dev version before this is closed. Also, the descriptions for each step in the interaction may need to be changed, but button style has been implemented.

terracoda commented 8 years ago

Sounds good. Though I think a shorter label for the Balloon is better.

<button>Grab ‪Yellow Balloon‬</button>

The phrase, go play with the Balloon would be good at the end of the Scene Summary. Also, please add commas in between WASD letters, remove "grab" from the directions in the navigation, and add "Space to release". Also, do you think it would be wise to remove the Help interaction, change it to something else like "Press H for Balloon Hot keys", or wait until I give you proper details?

<div tabindex="0" role="application" id="balloon-17-42-45-278-351" draggable="true" aria-grabbed="true" class="Balloon" aria-labelledby="balloon-label-17-42-45-278-351" aria-describedby="balloon-description-17-42-45-278-351 navigation-description-17-42-45-2i 78-351">
    <h3 id="balloon-label-17-42-45-278-351">‪Yellow Balloon‬</h3>
    <p aria-live="assertive" id="balloon-description-17-42-45-278-351">‪Yellow Balloon has a ‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪net negative‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬ charge, ‪several‬ more negative charges than positive ones.‬</p>
    <p id="navigation-description-17-42-45-278-351">‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪‪Use W, A, S, and D key to drag balloon up, left, down, and right. Spacebar to release. Press question mark for keyboard commands and help.‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬</p>
</div>
jessegreenberg commented 8 years ago

Great ideas @terracoda. I will create a checklist for each item mentioned in the comment above so that you can view progress. Though I will wait to add 'Go play with balloon' to end of screen summary until we discuss that further.

Also, do you think it would be wise to remove the Help interaction, change it to something else like "Press H for Balloon Hot keys", or wait until I give you proper details?

Yes, this is a quick change so I will go ahead and do this, though I think we should discuss some details further in addition. https://github.com/phetsims/balloons-and-static-electricity/issues/122 has initial discussion about this.

jessegreenberg commented 8 years ago

Checklist done. Found a new issue with the balloon interaction. The balloon no longer is 'dropped' on blur event.

jessegreenberg commented 8 years ago

Fixed in commit above.

terracoda commented 8 years ago

@jessegreenberg Sorry, I don't know what a blur event is :-) Sounds like this was fixed?

jessegreenberg commented 8 years ago

The 'blur' event occurs whenever the element loses focus. Indeed, this was fixed. Here is a version with this interaction:

http://www.colorado.edu/physics/phet/dev/html/balloons-and-static-electricity/1.2.0-accessible-instance.15/balloons-and-static-electricity_en.html?accessibility

jessegreenberg commented 8 years ago

Above commit is a new implementation of this interaction. Rather than switch focus to a new 'application' div, the role changes, and aria-live polite alerts describing the balloon or button are queued once the balloon is picked up and released.

It should work as long as the screen reader supports aria-live, role=application, role=button.

jessegreenberg commented 8 years ago

I just found this line in the ARIA 1.1 working draft:

Roles are element types and will not change with time or user actions. Role information is used by assistive technologies, through interaction with the user agent, to provide normal processing of the specified element type.

Unfortunately, this means that the above solution goes against specification.

terracoda commented 8 years ago

@jessegreenberg, I agree it seems hack to change element roles on the fly. I just read a couple of articles on Tabbed widgets (see #152). Is it possible to just change the focusability (change tabindex) of the balloon rather than hiding it all together. I'm just wondering if that would help make the description of the balloon naturally accessible via the cursor keys. I would have to adjust the description to make the user aware that they would have to grab it with grab balloon button before they can drag it.

Does this seem logical?

terracoda commented 8 years ago

Oops posted a little early. Here's an example using the Tab key:

  1. Grab Yellow Balloon, button
  2. Remove Wall, button
  3. Two-ballon experiment, Switch, Off
  4. etc.

With the down cursor key, starting with Grab Balloon button:

  1. Grab Yellow Balloon, button.
  2. Get ready to drag balloon. Once grabbed, press the W, A, S, or D key to drag up, left, down, or right. To let go, press Spacebar.
  3. Heading level 3, Yellow Balloon, not grabbed. Yellow Balloon is in centre of Play Area with a net neutral change. No more negative charges than positive ones. Use grab button to play with balloon.
  4. Heading level 3, Wall. Wall has a net neutral charge. No more negative charges than positive ones.

The words "not grabbed" would change to "grabbed" when the balloon is grabbed and the hotkey cue would have to be appended to the description.

So, the question is, is it possible to use tabindex and some dynamic content instead of role switching to create the 2 step interaction?

terracoda commented 8 years ago

@jessegreenberg, note new description for Grab Balloon button.

Adding the release cue description to the "on-demand help" of the button means that if the user never finds or bothers to check the hotkeys documentation, they will still have on-demand access to drag and release descriptions of the balloon interaction. And then we can shorten the grabbed balloon description.

I've updated the google sheets to reflect this change.

jessegreenberg commented 8 years ago

@terracoda, yes, I think your suggestion makes total sense, and I would expect a screen reader to support that kind of thing.

The purpose of switching roles rather than dynamically changing focus was to control the output of aria-live updates when the balloon is released. If the balloon is released, we want output like:

If focus goes to something other than the balloon, all we will hear is ' Grab balloon button'.

One option: when the balloon is released, do not focus back to 'Grab Balloon Button'. The user will have to go back to the grab button themselves, but this way we get to hear all of the live updates about the balloon. What do you think about this @terracoda?

If the user tabs something else while the balloon is being dragged, they will not hear any descriptions, but maybe that is acceptable.

As far as I am aware, there is no way to have a screen reader read live updates along with the description of the newly focused item. When something gets focused its description is assertive.

jessegreenberg commented 8 years ago

Maybe with the prototype screen reader we could try something like

Or we could come up with a different type of priority system.

We would be creating/changing aria attributes for the purpose of testing with the prototype. Not sure if we should do that yet.

terracoda commented 8 years ago

@jessegreenberg said:

One option: when the balloon is released, do not focus back to 'Grab Balloon Button'. The user will have to go back to the grab button themselves, but this way we get to hear all of the live updates about the balloon. What do you think about this @terracoda?

As long as the screen reader user can find the Grab Balloon button with the tab key, the cursor key and the the B key. I think this will be just fine. I think this might be closer to something more intuitive to a screen reader user.

jessegreenberg commented 8 years ago

Great, thanks @terracoda! I will give this a try.

jessegreenberg commented 8 years ago

Hmm, I forgot about the limitation in the PDOM that only one element representation can exist at a given time, and the peer must correspond to the correct node in the scene graph for the focus highlight to work correctly. It was working before because when one DOM representation for the balloon existed at a time.

jessegreenberg commented 8 years ago

We will need two scenery nodes to accomplish this. One for the 'Balloon Button' and another for 'draggable application div'. They will have identical bounds and locations. Dragging the balloon will drag both of these nodes. So BalloonNode can be the parent of both of these so that the drag handler extends to both children.

jessegreenberg commented 8 years ago

As long as the screen reader user can find the Grab Balloon button with the tab key, the cursor key and the the B key. I think this will be just fine.

Ah, except 'b' and cursor keys would be unusable because focus will still be on an 'application' element...

jessegreenberg commented 8 years ago

I wonder if we can force the order of events a different way? Lets say we track the previous active element. If previous active element is the draggable balloon div maybe we could then read all of the alerts after the description of the focused element has been read? This way it might sound like

User releases balloon

'Grab Balloon Button' is read first, but we should be able to hear everything else.

jessegreenberg commented 8 years ago

This won't quite work, 'focusin' event (used by cursor) seems to happen after 'focus' event (used by balloon). I expected the opposite!

jessegreenberg commented 8 years ago

How about a single 'aria-describedby' paragraph or list that contains all of the list items in https://github.com/phetsims/balloons-and-static-electricity/issues/133#issuecomment-224068292 (except for the balloon). The aria-describedby attribute would be shared by all buttons that can receive focus after the balloon is released. This way, we don't worry about alerts and aria-live at all. This description will only be read if the balloon was the last document.activeElement.

The output would be something like https://github.com/phetsims/balloons-and-static-electricity/issues/133#issuecomment-224068292

terracoda commented 8 years ago

@jessegreenberg said:

Ah, except 'b' and cursor keys would be unusable because focus will still be on an 'application' element...

I was thinking that the Grab Balloon button needs to be easily findable when the balloon is released, not when it is grabbed. Maybe I think I need to do some cognitive walk-throughs for this interaction, once I have a few more descriptions.

jessegreenberg commented 8 years ago

Sounds good @terracoda, walk-throughs would be helpful.

https://github.com/phetsims/balloons-and-static-electricity/issues/133#issuecomment-224082193 <--This works nicely, and might work well on all screen readers! This is much better than queuing multiple alerts.

The screen view listens to the 'blur' event and dispatches it down the DOM tree. If the target of the blur event is one of the application divs, the relatedTarget of the event gets an aria describedby which is a list of the 'messages that we would like to announce. On its focus, the screen reader will read the 'aria-describedby' list. The related target is stored and on its own blur, the 'aria-describedby' attribute is reset to its original value.

jessegreenberg commented 8 years ago

Works very well with the prototype screen reader. Does not work at all with NVDA + Firefox...

From the beginning of our work with the parallel DOM, we have been assuming that the document can be dynamic. I am beginning to have concerns that screen readers largely do not support adding/removing elements from the document and changing their attributes. I believe this is mostly due to the way screen readers use the Accessibility API, and generate a static accessibility tree separate from the HTML. It is assumed that most content will not be added to the document after it has loaded. This may be why we are having so much trouble with application-like content.

jessegreenberg commented 8 years ago

Perhaps this is because of the structure of the description content, I will try simplifying it to see if it works in a very basic case.

jessegreenberg commented 8 years ago

A single paragraph element works with NVDA + Chrome, but not on NVDA + Firefox. Perhaps the accessibilith API used by Firefox has more limitations with dynamic DOM element attributes.

jessegreenberg commented 8 years ago

In https://github.com/phetsims/balloons-and-static-electricity/issues/151, @jobara recommended a brilliant way to get aria-live working in a predictable way. This means that we can revert to using alerts rather than changing the aria-describedby id's to notify the user of this content.

jessegreenberg commented 8 years ago

All alerts in BASE should now use this method of updating live content.

jessegreenberg commented 8 years ago

Can these alerts be hidden or have 'aria-hidden'? I am not sure if this content should be found by the cursor.

jessegreenberg commented 8 years ago

With NVDA, releasing the balloon will will cause focus to go to the 'grab yellow balloon' button, but since the spacebar is down, the newly focused button will be clicked immediately and grab the balloon again.

jessegreenberg commented 8 years ago

We could release the balloon on keyup?

jessegreenberg commented 8 years ago

Yep, that works :+1:

jessegreenberg commented 8 years ago

We are definitely using this pattern. Closing for now, as new bugs are found additional issues will be created.