search

phetsims / balloons-and-static-electricity

"Balloons and Static Electricity" is an educational simulation in HTML5, by PhET Interactive Simulations.
http://phet.colorado.edu/en/simulation/balloons-and-static-electricity
GNU General Public License v3.0
6 stars 10 forks source link

Investigate problems role="application" #116

Closed jessegreenberg closed 8 years ago

jessegreenberg commented 8 years ago

We need to investigate the aria application role. Our understanding is that this role should allow all key events to be passed to the web application. This has not been our experience. The AT recognizes and announces the role, but does nothing with that information and continues to use 'a 'document' mode.

It is unclear is we are using the role incorrectly, or if something about the Parallel DOM context is preventing the role from being recognized. The application role has been out for a while so it seems unlikely that it is broken.

We may have alternate solutions for this simulation, but it would be good to identify why this role isn't working as we will inevitably use it in the future.

jessegreenberg commented 8 years ago

From W3: https://www.w3.org/TR/wai-aria/roles#application

When the user navigates an element assigned the role of application, assistive technologies that typically intercept standard keyboard events SHOULD switch to an application browsing mode, and pass keyboard events through to the web application. The intent is to hint to certain assistive technologies to switch from normal browsing mode into a mode more appropriate for interacting with a web application; some user agents have a browse navigation mode where keys, such as up and down arrows, are used to browse the document, and this native behavior prevents the use of these keys by a web application.

jessegreenberg commented 8 years ago

The following is an example of accessible drag and drop from Open Ajax: http://oaa-accessibility.org/example/17/

This example uses the application role. But the game also uses a grid role. My understanding from W3 is that this role on its own allows for arrow key navigation:

Grids allow the user to move focus between cells using two dimensional navigation.

I would like to test the above axample with a screen reader and with the application role removed. My guess is that it should work as it does now, without the announcement of 'application' from the screen reader.

jessegreenberg commented 8 years ago

The next test would be to listen for events with the application role in a Prototype HTML context, removed from scenery.

jessegreenberg commented 8 years ago

Here is one example of the application role used for the balloon in balloons-and-static-electricity. The following is the representation in the parallel DOM.

<div id="balloon-container-14-35-37-270-344" aria-live="polite" role="application" aria-labelledby="balloon-label-344" aria-describedby="balloon-description-344" >
    <div tabindex="0" id="balloon-14-35-37-270-344-345-348" draggable="true" class="Balloon" aria-labelledby="balloon-label-344" aria-describedby="balloon-description-344"></div>
    <h3 id="balloon-label-344">‪Yellow Balloon‬</h3>
    <p id="balloon-description-344">‪Yellow Balloon has a ‪neutral‬ charge, ‪no‬ more negative charges than positive ones.  Select WASD key to grab the balloon and start dragging. Select Tab for next item. Select question mark for keyboard help.‬</p>
</div>
jessegreenberg commented 8 years ago

I am confused by the example from Open Ajax. When removing the application role from the widget, the event goes on to the browser, but the X's and O's do not move to the board.

jessegreenberg commented 8 years ago

I just gave a test of the application role with the WASD keys. With the WASD keys, the application role works surprisingly well on most browsers! Here are my results.

For this test, I used the following html to represent the balloon:

<div id="balloon-container-14-39-42-275-349" aria-live="polite" role="application" aria-labelledby="balloon-label-349" aria-describedby="balloon-description-349" data-at-shortcutkeys="{'j': 'Key to start jumping', 'w': 'Key to jump to wall', 's': 'Key to jump to sweater'}">
    <div tabindex="0" id="balloon-14-39-42-275-349-350-353" draggable="true" class="Balloon" aria-labelledby="balloon-label-349" aria-describedby="balloon-description-349"></div>
    <h3 id="balloon-label-349">‪Yellow Balloon‬</h3>\
    <p id="balloon-description-349">‪Yellow Balloon has a ‪neutral‬ charge, ‪no‬ more negative charges than positive ones.  Use WASD keys to grab and drag balloon up, left, down and right. Tab for next object. Press question mark for keyboard commands and help.‬</p>
</div>

Here are my testing results using the WASD keys for drag and drop behavior:

Other than Chrome and Edge with NVDA, the application role seems to work for the WASD keys!

jessegreenberg commented 8 years ago

I wanted to double check the application role with the arrow keys. The following test uses the same HTML as above, but with the arrow keys replacing the WASD keys in the event listener.

jessegreenberg commented 8 years ago

So it looks like our main issue is with the arrow keys!

Back to the quote from W3:

some user agents have a browse navigation mode where keys, such as up and down arrows, are used to browse the document, and this native behavior prevents the use of these keys by a web application.

Looks like we can (generally) get the behavior we want for accessible drag and drop as long as we avoid the arrow keys.

jessegreenberg commented 8 years ago

With this information, I am inclined to avoid the text input hack suggested in issue #108. For now, I will leave the balloon represented as a div. We need to determine the best role for the balloon so that it is not announced as a section. For now, I will use a button role until further discussion.

jobara commented 8 years ago

@jessegreenberg i wonder if you'd consider supporting both the arrow keys and the letter keys. In that way the non-screenreader keyboard user could still use the arrow keys, and the screenreader user would also have a working keyboard solution.

We did that for our reordered component. http://build.fluidproject.org/infusion/demos/reorderer/gridReorderer/

Here you can use the arrow keys or the I,J,K,M keys. Don't worry so much about which letter keys we chose. It had to do with keys that weren't already used at the time, but it's fully configurable for an implementor to change.

jessegreenberg commented 8 years ago

That is a great suggestion @jobara, this is something we can do. I discussed this with @emily-phet and @terracoda and there was some concern that screen reader users and non-screen reader users would have a different experience, especially in collaboration. If I understood correctly, there should eventually be a visual cue to non-screen reader users that the WASD keys can be used to drag the balloon.

jessegreenberg commented 8 years ago

That being said, perhaps we should discuss again. I don't particularly see any harm in enabling both arrow and WASD keys for convenience to non - screen reader users.

jessegreenberg commented 8 years ago

That is great example @jobara, thank you for sharing! Having both methods of interaction available makes it robust and convenient.

With the application role, sometimes arrow keys work just enough with the screen reader to pick up the balloon, but not to drag it. It looks like in some cases the keyup event is fired immediately after keydown for the arrow keys. Enabling both in this case could make the simulation seem broken when arrow keys are not fully supported.

I should note that arrow keys and WASD keys always work perfectly if the user forces the screen reader into 'forms' mode (or whatever that mode is for the AT).

terracoda commented 8 years ago

@jessegreenberg, I agree with @jobara that supporting both letter keys and arrows is a good idea. I haven't given up on arrow keys and only thought the WASD keys were a temporary work around while we figured out how to best represent the balloon in code. I certainly see the balloon as a drag and drop, always have. I was more concerned about complicated interactions with the kinds of examples I seen.

The example above is a much simpler drag and drop interaction than other drag and drops I've seen. It's actually the kind of interaction I was envisioning when I brought up the menu role idea as a possible solution to get the arrow keys to work.

And I also agreed with @jobara that the menu role was not ideal, and that a deeper look at how to make a simple drag and drop was preferred. Thank you @jobara for sharing. I didn't realize you had already built a simple drag and drop.

In the case of the balloons, I feel one needs an explicit release (or drop) because the balloon doesn't necessarily stay where it is dropped. @jobara is the explicit drop in the above interaction just releasing the Control key?

Can we have a drag and drop without large gridcells? When the balloon is released it won't necessarily end up resting neatly in a grid cell. If the grid cells can be the small steps we have now, we may have a solution.

Giving the user the option to choose their own keys is a great idea. For right-handed folks IJKL may work better than WASD. In my last 3 interviews all 3 users had trouble understanding what WASD was, obviously none of them are gamers. Again @jobara, great suggestions.

terracoda commented 8 years ago

@jessegreenberg Also, nice bunch of testing above. Great discussion and results over all!

terracoda commented 8 years ago

@jessegreenberg @jobara In today's interview with Windows 10 and IE11 (Surface Pro 3 tablet with keyboard) the WASD keys worked sometimes and sometimes they didn't. In a few interactions, only every other key press moved the balloon. And in some cases, there was no movement and then the next press sent the balloon from the the bottom of the sweater all the way to the top in one go.

Some general things: User kept getting stuck on the main "reset" button - no wrapping to top. Will look more closely at video. Not sure what is causing that.

The Keyboard commands and help dialog only works when balloon has focus. Ideally help should be available from anywhere. I think this was a temporary implementation of help.

Dialog issue: the user ended up back in the content somehow without closing the dialog.

User didn't like that the word bullet was readout by JAWS in the keyboard dialog lists. I'll have to see if there is a way to suppress that naturally. I wonder if that is happening because the content in the dialog has document role or if it is a JAWS detail setting?

This user used a lot of his JAWS hotkeys.

jobara commented 8 years ago

@terracoda in regards to the drag and drop of the grid reorder demo. Yes the "ctrl" is what grabs and releases the item. This demo is specifically about moving an item around a grid. I believe you could have the drop targets be anything really. We have a few other demos as well list, image, and layout. The layout demo might be the most interesting to look at, as the drop targets depend on how many panels are in each column. Also note that in all of these examples, the drop target isn't necessarily where the draggable item is dropped. In these examples we incase the "target" size by actually picking the closest target to where the item is actually released. Hopefully this will be helpful for thinking about interactions for balloons and static electricity.

@terracoda in regards to the issues you saw on the Surface Pro 3 when user testing, I wonder if they are performance related. I have a Surface here running Windows 10, maybe we could try to replicate what you saw, when you are in Toronto.

terracoda commented 8 years ago

@jobara thanks for more links to examples. The interaction I originally proposed is conceptually a drag and drop: one key to grab (and drag) and one key to drop (release). The only issue was technically implementing the simple interactions. And we got side tracked by application role and screen reader modes. This conversation has brought us back around to the essentials and this is great.

Re the delayed dragging on the Surface Pro 3, I agree that it is likely performance related. I haven't seen such behaviour on any other computer. Yes, let's test things when I am in Toronto. I will be at the IDRC Feb 1st to Feb 5th.

jessegreenberg commented 8 years ago

More excellent examples @jobara. These are very helpful, thank you!

I still think the application role will be very important for us. For example, I can't get these examples to work unless the screen reader is forced into 'forms' mode, which is exactly the issue we are facing in Balloons and Static Electricity. It seems like a new question is: How important is it that the screen reader toggle to 'forms' mode automatically? Would it be acceptable to explicitly tell the user to switch modes?

terracoda commented 8 years ago

@jessegreenberg I agree that this is a good question. We are investigating what things can be done implicitly and what has to be explicit. Usually switching to forms mode is an implicit thing that happens and the AT tells the user it has happened, usually with a pop (JAWS makes a popping sound).

terracoda commented 8 years ago

By the way, the whatsock website by Bryan Garaventa has had a re-design. It is much easier to read now.

terracoda commented 8 years ago

@jessegreenberg I think what we need is an

Unassociated Drag and Drop.

See section 2.5 of whatsock website.

jessegreenberg commented 8 years ago

I agree @terracoda, the optimal solution would involve allowing the transition to occur implicitly.

terracoda commented 8 years ago

@jessegreenberg, quoting Whatsock:

The first thing to note, is that the attributes aria-grabbed and aria-dropeffect have no functional effect on the draggability of a component. This must be programmed in advance using JavaScript and CSS to handle mouse and keyboard interaction. The drag and drop ARIA attributes simply convey these relevant behaviors to Assistive Technologies.

My question, is it possible we are not "conveying" the the relevant behaviours correctly?

My initial observations on comparison the with Whatsock code. In terms of roles, there is no talk of the application role for a draggable object, but a draggable object "must have a valid role". Based on a comparison of the currently implemented code and the code from Whatsock, we may have 3 issues:

  1. A valid role on the actual draggable object. Whatsock suggests role="button".
  2. A missing aria-grabbed="false" attribute. The draggable="true" attribute is an HTML5 attribute, not an ARIA attribute. I'm not sure, but I've read elsewhere that screen readers began to support aria attributes before HTML5 semantics. According to Whatsock, the aria-grabbed attribute "should be added to either the draggable container element, or to the drag handle element, whichever is designed to be focusable."
  3. The draggable div wrapper is not wrapping the actual balloon content, which doesn't look right to me.

Taking your code and the Whatsock code, we might try the following:

<div class="Balloon" id="balloon-14-39-42-275-349-350-353" tabindex="0" role="button" draggable="true" aria-grabbed="false" aria-labelledby="balloon-label-349" aria-describedby="balloon-description-349">
<h3 id="balloon-label-349">Yellow Balloon</h3>
<p id="balloon-description-349">Yellow Balloon has a ‪neutral‬ charge, ‪no‬ more negative charges than positive ones.  Use WASD keys to grab and drag balloon up, left, down and right. Tab for next object. Press question mark for keyboard commands and help.</p> 
</div>

@jessegreenberg, given what you've tried thus far, does this code fill any gaps?

jessegreenberg commented 8 years ago

@terracoda, thank you for going through this resource. This code does fill some gaps. We have not tried placing the draggable and aria-grabbed attributes on the container element. I will give this a try and see how it behaves without the application role for the same platforms tested above.

terracoda commented 8 years ago

Great! And I'm not sure what you are doing with the class="balloon", but Whatsock says all the functionality has to

be programmed in advance using JavaScript and CSS to handle mouse and keyboard interaction.

jessegreenberg commented 8 years ago

I believe that is something that we can ignore in this context. All functionality is programmed in advance with JavaScript and Scenery, which does not need to reference the class.

terracoda commented 8 years ago

In section 2.4 on, Whatsock discusses techniques for binding CSS with ARIA Roles and States. This technique might be helpful for tying together some aspects of the content layer and the visual layer - not sure, though.

jessegreenberg commented 8 years ago

Very cool, thank you @terracoda. I will take a look after this.

terracoda commented 8 years ago

@jessegreenberg regarding the movement of balloon and a grid, since in practice the balloon isn't really moving on a grid, do you think it is possible to use the current moving model to identify a set of drop targets that fill the Play Area with an invisible grid?

terracoda commented 8 years ago

@jessegreenberg , I just noticed, the Live Demos section on Whatsock (http://whatsock.com/). There is drag and drop, but it is quite different from our interaction.

jessegreenberg commented 8 years ago

@terracoda, the current model does not include this, but we could add a grid structure of possible drop targets. We should discuss this further, I am not quite sure what you have in mind for design and how this would work with the current balloon interaction to say for sure.

Thanks for pointing out that example. It is a very different interaction, but still useful. Unfortunately, no keyboard dragging with arrow or letter keys.

I am very curious about the reference to the 'AccDC API'. From whatsock, it is a

free development resource for international businesses, organizations, and academic institutions wishing to incorporate Accessible Innovation within future web technologies.

This is interesting, I would like to investigate this further.

See https://github.com/phetsims/scenery/issues/509.

terracoda commented 8 years ago

@jessegreenberg, great you started an issue for the AccDC API.

jessegreenberg commented 8 years ago

Another example of drag and drop from whatsock using AccDC API:

http://whatsock.com/tsg/Coding%20Arena/Drag%20and%20Drop/demo.htm

Again, this example only uses tab and 'enter'. The animated drag and drop behavior are not controlled by the user.

jessegreenberg commented 8 years ago

The above example is the closest in behavior to what we are attempting. Here is the interaction interface for accessible drag and drop in that example:

Press the Tab and Shift+Tab keys to switch between items.
Press the Enter key to initiate drag.
Mouse users can drag and drop the desired item as usual.

After inspecting the html, I notice the follodwing:

Draggable elements are contained in lists with the following html:

<div id="AccDC1453918598522" aria-grabbed="false" style="display: inline; position: absolute; z-index: 1;">
  <div style="position: absolute; clip: rect(1px 1px 1px 1px); padding: 0px; border: 0px; height: 1px; width: 1px; overflow: hidden; z-index: -1000;"></div>
  <div id="AccDC1453918598544" style="display: inline;">
    <img data-label="The Last Unicorn by Peter S. Beagle" data-price="8.93" src="img/TheLastUnicorn.jpg" alt="The Last Unicorn by Peter S. Beagle ($8.93)" title="The Last Unicorn by Peter S. Beagle ($8.93)">
    <a href="#" class="ddLink" style="clip: rect(1px 1px 1px 1px); border: 0px; position: absolute; z-index: -1000; padding: 0px; height: 1px; width: 1px; overflow: hidden;">Move The Last Unicorn by Peter S. Beagle to Cart</a>
    </div>
  <div style="position: absolute; clip: rect(1px 1px 1px 1px); padding: 0px; border: 0px; height: 1px; width: 1px; overflow: hidden; z-index: -1000;"></div>
</div>

To announce that an element is being 'dragged' (animated across the screen), the following aria-live region is used in a div

<div aria-live="polite" style="position: absolute; clip: rect(1px 1px 1px 1px); padding: 0px; border: 0px; height: 1px; width: 1px; overflow: hidden; z-index: -1000;"></div>

It looks like 'dragging' announcements will not come naturally from an aria-role. This is something we will have to announce with a live region. In addition, the only role used in this example is 'aria-grabbed'

terracoda commented 8 years ago

@jessegreenberg We already use Tab and Shift+Tab between the balloons, so that is consistent with our application. Also, depending on how a user gets to the balloon, the Enter key is used to activate it. I saw this behaviour with P5. He used the JAWS hotkey for Regions to navigate to the Balloon application Region. He immediately heard that he was in forms mode (from JAWS), then hit Enter to enter the form and then he continued.

In the example above, the books are dragged by the application, not by the user, so the arrows are not needed in this example. I think the important detail is that the aria-grabbed is on the actual draggable element. I haven't listened to this interaction.

I wouldn't expect any dragging announcement to be made, but I would expect an announcement from AT about when the state changes from ungrabbed to grabbed, thus ready for dragging.

terracoda commented 8 years ago

@jessegreenberg, also what is important about the items being contained in a list is that by being in a list, they have the implicit role "listitem"! According to the ARIA documentation an element must have a valid role in order to be able to use aria-grabbed and aria-drop-effect.

For us, the role button should work, but in theory, even the role image should work. In the specification, there is no mention of forms mode and application role.

It's possible that screen readers do not recognize aria-grabbed and aria-dropeffect, but if they do and we have communicated that information correctly, they should give up the use of the arrow keys to the application - at least that's how I understand all the information I read recently. And of course, you have to make the arrow keys (and other letter keys) work with your magic.

jessegreenberg commented 8 years ago

@terracoda I am going to create another issue for your comments in https://github.com/phetsims/balloons-and-static-electricity/issues/116#issuecomment-174755489

There are some issues there that I would like to discuss further.

jessegreenberg commented 8 years ago

He used the JAWS hotkey for Regions to navigate to the Balloon application Region. He immediately heard that he was in forms mode (from JAWS), then hit Enter to enter the form and then he continued.

Very interesting! That is good to know, thanks @terracoda.

In the example above, the books are dragged by the application, not by the user, so the arrows are not needed in this example.

Yes, that is true. These examples do not require the arrow keys for functionality. We have yet to encounter an example where the arrow keys are used to drag an element from one location to another outside of a grid.

I wouldn't expect any dragging announcement to be made, but I would expect an announcement from AT about when the state changes from ungrabbed to grabbed, thus ready for dragging.

That is what I expected too! But that does not seem to happen. When I navigate to a book in Bryan Garaventa's example in with NVDA and JAWS, I hear something like this:

"Move The Last Unicorn to cart draggable droppable"

If you navigate away from the book and then back to it while it is still being dragged, the screen reader announces something like:

"Move The Last Unicorn to cart droppable"

I think that is why the aria-live region for 'dragging' is necessary. It seems like the aria-grabbed and aria-dropeffect provide information upon navigation. No update is announced about the state of the object.

jessegreenberg commented 8 years ago

what is important about the items being contained in a list is that by being in a list, they have the implicit role "listitem"! According to the ARIA documentation an element must have a valid role in order to be able to use aria-grabbed and aria-drop-effect.

Yes that is a good point. Using a valid role in combination with aria drag and drop attributes will be important.

jessegreenberg commented 8 years ago

I recently tested with the balloon represented by the following HTML:

<div tabindex="0" role="button" id="balloon-14-35-38-271-344-347" draggable="true" aria-grabbed="false" class="Balloon" aria-labelledby="balloon-label-14-35-38-271-344-347" aria-describedby="balloon-description-14-35-38-271-344-347" aria-dragged="false">
    <h3 id="balloon-label-14-35-38-271-344-347">‪Yellow Balloon‬</h3>
    <p id="balloon-description-14-35-38-271-344-347">‪Yellow Balloon has a ‪neutral‬ charge, ‪no‬ more negative charges than positive ones.  Use WASD keys to grab and drag balloon up, left, down and right. Tab for next object. Press question mark for keyboard commands and help.‬</p>
</div>

I tested with the same platforms with both WASD and Arrow keys. The aria-grabbed makes it so that 'draggable' is read in almost every screen reader. But keyboard dragging only worked for one platform. Here is the summary of test results:

Testing WASD keys: NVDA + Chrome: 'Draggable' is read in description, WASD keys do not work

NVDA + Firefox: 'Draggable' is read as part of description, WASD keys do not work

NVDA + IE: 'Draggable' read at very end of description, WASD keys do not work

NVDA + Edge: 'Draggable' NOT read at end of description, WASD keys donot work

JAWS + Chrome: 'Draggable' read twice, WASD keys do not work

JAWS + Firefox: 'Draggable' read twice, WASD keys do not work

JAWS + IE: 'Draggable' read twice, WASD keys do not work

JAWS + Edge: 'Draggable' NOT read, WASD keys Work. Many other descriptions were not functional

Testing Arrow Keys: JAWS + Chrome: 'Draggable' Read, arrow keys do not work

JAWS + Firefox: 'Draggable' Read, arrow keys do not work

JAWS + IE: 'Draggable' Read, arrow keys do not work

JAWS + Edge: 'Draggable' NOT read, arrow key event passed to browser, but 'keyup' event fired immediately by accessibility API Drag and drop does not work

NVDA + Chrome: 'Draggable' read, arrow keys do not work

NVDA + Firefox: 'Draggable' read, arrow keys do not work

NVDA + IE: 'Draggable' read, arrow keys do not work

NVDA + Edge: 'draggable' NOT read, arrow keys do not work

jessegreenberg commented 8 years ago

By switching the role to application, we have the following html representation for the balloon:

<div tabindex="0" role="application" id="balloon-14-35-38-271-344-347" draggable="true" aria-grabbed="false" class="Balloon" aria-labelledby="balloon-label-14-35-38-271-344-347" aria-describedby="balloon-description-14-35-38-271-344-347" aria-dragged="false">
    <h3 id="balloon-label-14-35-38-271-344-347">‪Yellow Balloon‬</h3>
    <p id="balloon-description-14-35-38-271-344-347">‪Yellow Balloon has a ‪neutral‬ charge, ‪no‬ more negative charges than positive ones.  Use WASD keys to grab and drag balloon up, left, down and right. Tab for next object. Press question mark for keyboard commands and help.‬</p>
</div>

With this representation, I observe the same results as https://github.com/phetsims/balloons-and-static-electricity/issues/116#issuecomment-174599496 and https://github.com/phetsims/balloons-and-static-electricity/issues/116#issuecomment-174601771

terracoda commented 8 years ago

@jessegreenberg Are you sure aria-dragged is a valid aria attribute? I can only find it on the MDN page, not in the spec.

terracoda commented 8 years ago

Also, @jessegreenberg that's a lot of testing and sadly not a lot of working keys :(

jessegreenberg commented 8 years ago

Are you sure aria-dragged is a valid aria attribute? I can only find it on the MDN page, not in the spec.

Great catch @terracoda! Totally my mistake. I initialized aria-grabbed correctly, but accidentally set aria-dragged when linking to the model.

I don't think that this will have changed the screen reader behavior too much, but I will fix and retest on the platforms listed above.

jessegreenberg commented 8 years ago

Also, @jessegreenberg that's a lot of testing and sadly not a lot of working keys :(

Yes, unfortunately :(. Our best results so far involve the application role and the WASD keys. Application role and arrow keys work, but only with JAWS + IE.

jessegreenberg commented 8 years ago

I just tested this. The interaction with the keyboard behaves exactly as it does above. However, there is one difference:

For NVDA + Firefox with role=application, NVDA announces when the balloon toggles from aria-grabbed=true to aria-grabbed=false in exactly the way that @terracoda suggested in https://github.com/phetsims/balloons-and-static-electricity/issues/116#issuecomment-175835539

This does not happen for any other browser or with JAWS.

terracoda commented 8 years ago

Well, that's good that at least one browser/AT combo announces the state. As we learned today from Joseph, aria-grabbed and aria-dropeffect have not been implemented well.

jessegreenberg commented 8 years ago

It seems like either the browsers or the AT are not handling the application role correctly. There are two steps to follow through to see if we can get the desired keyboard navigation:

terracoda commented 8 years ago

@jessegreenberg I agree that we need to investigate the the Accessibility Tree and simplify the context. Would you mind posting the rendered HTML as it is in Instance 11? I know it is slightly different from my prototyped PDOM.

Here are the few approaches we might take based on discussions with Joseph Schuehammer and some post interview thoughts.

  1. Reduce nesting of code structure. Doing this should reduce the duplication of content. This might involve not using the H3 and aria-labelledby structure for the Balloon (not 100% sure, though).
  2. Alternatively, we can try using a span element instead of a div as a container. A span is an inline element (not a block element) and may behave differently with the role application than a div.
  3. We can experiment with input type=number (which can be associated with text strings). The Balloon does indeed have a value (a net zero or net negative charge), so input type=number works well semantically. This may also have the added benefit of addressing a description issue that some participants raised in the interviews. With this input type, I think we briefly looked at the idea of using aria-role=spinbox.
  4. One other idea is to change the interaction flow by adding an explicit grab before the drag. I think this was my original idea way back when. A comment from one of the participants supports this interaction pattern. He mentioned that in gaming, he often uses the Spacebar to pickup AND drop things. Would a change in the design of the interaction successfully push the screen reader into forms mode and then the application can deal with the drag interaction after that.
  5. Finally, need to look into custom labeling possibilities with the the ARIA specification. Can we tell a screen reader to read out the balloon as simply as, "Interactive Balloon" using an ARIA technique?

Example with explicit grab interaction (number 4):

So, I am wondering if an explicit grab might improve the interaction for keyboard (and screen reader users) even though it is an extra key press? The reason behind it is that the user would get the instructions for the interaction in two steps which might be more understandable.