When using keyboard nav rectangular focus box sometimes disappears when moving balloon

[Nancy-Salpepi]

Test device Mac11 (M1 chip)

Operating System 11.4

Browser safari 14.1.1

Problem description https://github.com/phetsims/qa/issues/689

Sometimes when switching between using the trackpad and keyboard nav, the dotted/solid rectangular box disappears from around the balloon that I am moving.

Steps to reproduce It doesn't happen every time with these steps, but I can't figure out what I'm doing differently to change them:

1. Go to preferences>visual tab to enable interactive highlights
2. Using the trackpad, remove wall and select 2 balloons
3. Using the trackpad, click on the reset balloon button
4. Hover the pointer over the yellow balloon so that the focus box is around it.
5. Press the tab button. The box with "The Space Bar Icon to Grab or Release" should appear
6. Press the space bar and use the left arrow to move the balloon towards the sweater.
7. If it doesn't happen the first time, try repeating steps 3-6.

Visuals https://drive.google.com/file/d/1JVoXtHJY-D1waFixCZc0DrCvnxJYgiSH/view?usp=sharing

Troubleshooting information: !!!!! DO NOT EDIT !!!!! Name: ‪Balloons and Static Electricity‬ URL: https://phet-dev.colorado.edu/html/balloons-and-static-electricity/1.5.0-dev.35/phet/balloons-and-static-electricity_all_phet.html Version: 1.5.0-dev.35 2021-08-06 21:46:41 UTC Features missing: touch Flags: pixelRatioScaling User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/605.1.15 Language: en-us Window: 1440x683 Pixel Ratio: 2/1 WebGL: WebGL 1.0 GLSL: WebGL GLSL ES 1.0 (1.0) Vendor: WebKit (WebKit WebGL) Vertex: attribs: 16 varying: 31 uniform: 1024 Texture: size: 16384 imageUnits: 16 (vertex: 16, combined: 32) Max viewport: 8192x8192 OES_texture_float: true Dependencies JSON: {}

[jessegreenberg]

Thanks @Nancy-Salpepi, I am responsible for interactive highlights, I'll take a look at this.

[jessegreenberg]

Thanks @Nancy-Salpepi - I don't think this is a bug but a result of two factors of the Interactive Highlight design. 1) Only one highlight is shown at a time. 2) The "Interactive highlight" looks the same as the focus highight.

What is happening in the video capture is that the mouse is over one balloon to activate the Interactive Highlight. Then tab puts focus on the balloon and removes the interactive highlight. Then when the balloon moves from keyboard, the mouse is still over the OTHER balloon and the interactive highlight activates for that one. This removes the focus highlight. A more simple case of this can be seen here without balloon movement:

In this case, focus is always on the green balloon, but the interactive highlight adds and removes the focus highlight. When the mouse leaves the balloon, we restore the focus highlight.

Over to @terracoda to review, do you agree this is behaving as we expect?

[terracoda]

Currently, the interactive highlight does not recognize the two modes grabbed (dashed outline) and not grabbed (solid outline).

Hopefully, once the interactive highlight behaviour recognizes these 2 modes this issue will be resolved.

We will work on improving the interactive highlight in our work on Friction. We can do a maintenance release for BASE when we publish Friction with Voicing and Interactive Highlights.

[terracoda]

@jessegreenberg can you put the correct labels to make it clear this issue will addressed later?

[jessegreenberg]

If I understand correctly this will still happen after we work on #516, just making sure that is correct @terracoda. But happy to close.

[terracoda]

Thanks for the clarification @jessegreenberg, I misunderstood what was happening.

If the green balloon is grabbed and has focus, why does the grabbed-dashed-focus highlight disappear when hovering over the yellow balloon with the mouse?

Is it that with our current system of focus and interactive highlights, only one object can be highlighted at once?

[terracoda]

Actually, there is a problem and some very inconsistent behaviour.

When I grab a balloon and move it with the keyboard over an uncharged second balloon, or other interactive objects (buttons, etc), my grabbed-dashed-focus-highlight sometimes goes solid, or sometimes transfers to another interactive object (even though focus has not moved), and when that happens the dashed highlight on the grabbed balloon disappears. The solid highlight can move, even if focus has not moved, and even when I am not not actively using the mouse.

This is a bug. I have a .mov file I can share, but I think it is pretty easy to re-produce.

You need to have interactive highlights enabled and maybe make up at least one click with the mouse somewhere once or twice before or during keyboard use.

Once there is bit of mixed use, the highlight behaves inconsistently pretty regularly.

I am using Safari and MacOs 11.4

[terracoda]

@jessegreenberg, I also noticed once (having a hard time reproducing) that I got a hover style on the Reset Balloon button when I moved a balloon over it with the keyboard.

I am wondering if there is some confusion going on in the sim when interactive highlights are enabled and there is a bit of mixed input use.

Questions and thoughts on expected behaviour

1. Mouse clicks move active focus, right?
2. Keyboard focus should not move (unless there is a click away with the mouse, or Tab away, etc.) when a balloon is actively grabbed and being dragged using the keyboard (arrow keys, letter keys).
3. If a balloon is grabbed with the keyboard, the dashed highlight should remain until focus is lost and balloon is released. Hover over other controls can activate the hover interactive highlight, but the dahsed-grabbed-highlight should remain until focus is lost.
4. Moving a balloon with the keyboard or mouse should not ignite interactive highlights on other controls when the balloon passes over them.
5. Can our current system of highlights handle two things being highlighted at the same time? (see https://github.com/phetsims/balloons-and-static-electricity/issues/523#issuecomment-896844034)
6. I would expect that a balloon grabbed with the keyboard and thus having the dashed-grabbed-keyboard-focus highlight would keep that highlight even if I hover over the green balloon (other interactive element) without actually moving focus. Once focus is moved or lost, the balloon would be released and loose it dashed-grabbed highlight.

@jessegreenberg, let me know if you need to jump on a call.

[terracoda]

And my first questions in https://github.com/phetsims/balloons-and-static-electricity/issues/523#issuecomment-896844034 are relevant, too, so I will add them to list in the previous comment.

[Nancy-Salpepi]

Thanks @terracoda you worded that much better than I did. I noticed that same inconsistent behavior you commented on above.

[jessegreenberg]

@terracoda and I met today to discuss the behavior. What was happening for all cases is that when an interactive component moves under the mouse, regardless of what the state of the current highlight is a new highlight for the interactive component is made active and clears the previous one. This causes some confusing cases, particularly when the focus highlight indicates state like the ones in BASE.

For now, we decided to implement a rule that the Interactive Highlight cannot activate while the DOM focus highlight is already active. We may want to have a different behavior in the future (such as supporting multiple kinds of highlights being visible at a time) but this will be sufficient for now and get us through upcoming publications.

@Nancy-Salpepi can you please review the behavior now and confirm the behavior you noted is no longer possible? (change made to scenery, will require a pull before testing)

[Nancy-Salpepi]

Looks good! Dashed highlight remains on the balloon I am moving. It didn't switch to the other balloon nor did it oscillate between dotted and solid.

[jessegreenberg]

OK great, thanks! Closing.

[Nancy-Salpepi]

@jessegreenberg I noticed today that when moving the balloon using keyboard nav, the focus box will change from dotted line to solid line when it passes over area where mouse focus is (even though mouse pointer is not visible). Like before, this is when interactive highlights is on. Should I keep in this issue or open a new one?

[Nancy-Salpepi]

I can also see then even when I am not currently interacting with an object, when it moves over the mouse focus, the solid rectangle appears.

https://drive.google.com/file/d/1JM9z-rTbC76Z2MZxihJPduplBgUwiC3l/view?usp=sharing

[jessegreenberg]

Thanks @Nancy-Salpepi - If I understand correctly this is covered by the fix we made before. Is what you are describing still happening in master?

[Nancy-Salpepi]

My fault. Sorry about that @jessegreenberg

[terracoda]

In https://github.com/phetsims/balloons-and-static-electricity/issues/523#issuecomment-897034345 we have made a fix for this, but we will not apply this fix for this publication. The issue is less dramatic in this sim since there are no dashed grabbed highlights on any interactions in this sim.

In addition, the issue is an edge case requiring a user to have actively chosen "Interactive Highlights", and to use a mixed input method (both Keyboard and mouse), and to leave their mouse pointer on an interactive element when they switch to using the keyboard. While this is obviously not impossible, we feel it is rare enough that it shouldn't disrupt the user experience too much.

We will include this fix in the next maintenance release or re-publication for this sim.

[terracoda]

Oops moving comment to https://github.com/phetsims/john-travoltage/issues/448#issuecomment-901284337