Investigate problems role="application"

[jessegreenberg]

We need to investigate the aria application role. Our understanding is that this role should allow all key events to be passed to the web application. This has not been our experience. The AT recognizes and announces the role, but does nothing with that information and continues to use 'a 'document' mode.

It is unclear is we are using the role incorrectly, or if something about the Parallel DOM context is preventing the role from being recognized. The application role has been out for a while so it seems unlikely that it is broken.

We may have alternate solutions for this simulation, but it would be good to identify why this role isn't working as we will inevitably use it in the future.

[jessegreenberg]

Sure @terracoda, here is the up to date PDOM for BASE:

<div class="accessibility">
    <div class="ScreenView" id="screen-view-17-42">
        <header role="banner" aria-labelledby="scene-label-17-42">
            <h1 id="scene-label-17-42">‪Balloons and Static Electricity‬</h1>
        </header>
        <main class="ScreenViewContainer">
            <section>
                <h2>‪Scene Summary‬</h2>
                    <p>‪This simulation contains a Play Area and a Control Panel. The Play Area is a room. It has a balloon, a sweater and a wall. The Control Panel allows you to add a balloon, remove wall and reset entire experiment. Currently, sweater and wall have many pairs of negative and positive charges, the balloon, just a few. Balloon is in middle of Play Area, evenly between sweater and wall. Tab to navigate to next object. Press question mark for keyboard commands and help.‬</p>
            </section>
            <section id="play-area-17-42-45" aria-labelledby="heading-node-44">
                <div id="sweater-17-42-45-46" aria-live="polite" aria-labelledby="sweater-label-17-42-45-46" aria-describedby="sweater-description-17-42-45-46">
                    <h3 id="sweater-label-17-42-45-46">‪Sweater‬</h3>
                    <p id="sweater-description-17-42-45-46">‪The sweater has a ‪neutral‬ charge with ‪no‬ more positive charges than negative ones.‬</p>
                </div>
                <div tabindex="0" role="application" id="balloon-17-42-45-278-351-354" draggable="true" aria-grabbed="false" class="Balloon" aria-labelledby="balloon-label-17-42-45-278-351-354" aria-describedby="balloon-description-17-42-45-278-351-354 navigation-description-17-42-45-278-351-354">
                    <h3 id="balloon-label-17-42-45-278-351-354">‪Yellow Balloon‬</h3>
                    <p aria-live="assertive" id="balloon-description-17-42-45-278-351-354">‪Yellow Balloon has a ‪neutral‬ charge, ‪no‬ more negative charges than positive ones.‬</p>
                    <p id="navigation-description-17-42-45-278-351-354">‪Use W A S D keys to grab and drag balloon up, left, down and right. Tab for next object. Press question mark for keyboard commands and help.‬</p>
                </div>
                <div tabindex="0" role="application" id="balloon-17-42-45-278-279-282" draggable="true" aria-grabbed="false" class="Balloon" hidden="" aria-labelledby="balloon-label-17-42-45-278-279-282" aria-describedby="balloon-description-17-42-45-278-279-282 navigation-description-17-42-45-278-279-282">
                    <h3 id="balloon-label-17-42-45-278-279-282">‪Green Balloon‬</h3>
                    <p aria-live="assertive" id="balloon-description-17-42-45-278-279-282">‪Green Balloon has a ‪neutral‬ charge, ‪no‬ more negative charges than positive ones.‬</p>
                    <p id="navigation-description-17-42-45-278-279-282">‪Use W A S D keys to grab and drag balloon up, left, down and right. Tab for next object. Press question mark for keyboard commands and help.‬</p>
                </div>
                <div id="wall-17-42-45-164" aria-labelledby="wall-label-17-42-45-164">
                    <h3 id="wall-label-17-42-45-164">‪Wall‬</h3>
                    <p id="wall-description-17-42-45-164">‪The wall has a neutral charge, no more positive charges than negative ones.‬</p>
                </div>
            </section>
            <section id="control-panel-17-42-423" aria-labelledby="heading-node-424">
                <h2 id="heading-node-424">‪Control Panel‬</h2>
                <div id="element-container-440">
                    <button aria-describedby="441">‪Remove Wall‬</button>
                    <p id="441">‪Toggle to conduct experiments with or without the wall.‬</p>
                    <p id="442" aria-live="assertive" aria-hidden="true">‪Wall added to Play Area‬</p>
                </div>
                <div id="toggle-container17-42-423-473-467-454-449" aria-live="polite">
                    <button id="abswitch-17-42-423-473-467-454-449-453" aria-pressed="false">‪Two Balloon Experiment‬</button>
                    <p aria-hidden="true">‪Green balloon removed from Play Area‬</p>
                </div>
                <div id="element-container-465">
                    <button>‪Reset Balloon‬</button>
                    <p id="466">‪Select to ‪Reset Balloon‬ to initial position and an uncharged state.‬</p>
                </div>
                <input type="reset" value="‪Reset All‬" tabindex="0">
            </section>
        </main>
    </div>
</div>

[terracoda]

@jessegreenberg Thanks!

[jessegreenberg]

Reduce nesting of code structure.

From above, here is the representation of the balloon:

<div tabindex="0" role="application" id="balloon-17-42-45-278-351-354" draggable="true" aria-grabbed="false" class="Balloon" aria-labelledby="balloon-label-17-42-45-278-351-354" aria-describedby="balloon-description-17-42-45-278-351-354 navigation-description-17-42-45-278-351-354">
    <h3 id="balloon-label-17-42-45-278-351-354">‪Yellow Balloon‬</h3>
    <p aria-live="assertive" id="balloon-description-17-42-45-278-351-354">‪Yellow Balloon has a ‪neutral‬ charge, ‪no‬ more negative charges than positive ones.‬</p>
    <p id="navigation-description-17-42-45-278-351-354">‪Use W A S D keys to grab and drag balloon up, left, down and right. Tab for next object. Press question mark for keyboard commands and help.‬</p>
</div>

I am not sure how to reduce nesting here. Perhaps we can try it without descriptions and labels and see if the balloon behaves any better. We can then try to add descriptions and attributes back in to the balloon to see what is causing the repetition.

Alternatively, we can try using a span element instead of a div as a container.

I did not quite understand this, can a span exist on its own without being wrapped by another element? It seems strange that a widget would be represented by a span element. Also, I don't think inline elements can contain block elements like p.

We can experiment with input type=number (which can be associated with text strings).

I will give this a shot! We will see how browsers behave with this and how AT's read this element. My initial concern is that an input of type number that represents charge could be confusing when the user is actually controlling the position.

One other idea is to change the interaction flow by adding an explicit grab before the drag. Would a change in the design of the interaction successfully push the screen reader into forms mode and then the application can deal with the drag interaction after that. Example with explicit grab interaction (number 4):

user moves focus to balloon with the Tab key and hears "Play with Balloon, button". user presses Spacebar (or Enter) to play with the Balloon. Nav/interaction cue is read out. "Use W, A, S...to drag balloon up, left.... Press Spacebar to release."

Sure, we could add an explicit 'grab' interaction. A two step interaction could be helpful. When would the user hear the descriptions about the balloon charge/location?

[terracoda]

@jessegreenberg Regarding the nesting: that was likely caused by the form control that was rendered in earlier versions. I agree, I don't think we can reduce nesting - there is no nesting now :-)

Regarding duplication, as discussed today, this might be caused by the screen reader reading out the content once as aria-labelledby / aria-describedby and once as actual content. Not sure how to handle this problem. Maybe dynamically manage the attributes (labelledby and describedby), but that is just a guess.

Regarding a new two-step interaction, and when to read charge and position information, I have to think this through a little, but I think we'll need a super brief description upon grab, and then upon release something a little more verbose. The "Grab Balloon" button label would hopefully be read out last, after the release description.

[jessegreenberg]

https://github.com/phetsims/balloons-and-static-electricity/issues/116#issuecomment-177036081

After a recent update to firefox, Arrow keys are now supported by NVDA + Firefox on my Windows 10 Machine!! No change to the current balloon implementation.

In general, accessibility is performing much better on this platform after this update. I would like to know why.

[jessegreenberg]

Just tested with JAWS, this AT exhibits the same issues on all platforms. Event goes to the browser, but 'keyup' fired immediately after 'kedown' preventing dragging behavior.

[jessegreenberg]

Alternatively, we can try using a span element instead of a div as a container.

Replacing div with span does not seem to have an impact on screen reader behavior either way.

[jessegreenberg]

One other idea is to change the interaction flow by adding an explicit grab before the drag.

Creating a separate issue for this.

[jessegreenberg]

From https://github.com/phetsims/balloons-and-static-electricity/issues/116#issue-127963769

We need to investigate the aria application role. Our understanding is that this role should allow all key events to be passed to the web application. This has not been our experience.

We have learned a lot about the application role since ths issue was first opened. Primarily, we now understand that it is up to the AT to determine which events are passed to the browser and which ones aren't. This makes it more challenging to use certain keys for interactive behavior, but we can generally expect certain keys (such as WASD) to be handed to the browser with the application role. Our next task will be to submit a bug report to freedom scientific to understand why keydown events from the arrow keys are not supported with role=application. I will open a special issue in this repo to track this.

This issue has gotten a bit lengthy, and I think we have learned what we needed from the discussions and tests above. Closing, and further investigation with application role will be reduced to more specific issues.