fluid-lab / gamepad-navigator

GSoC 2020 project
Other
6 stars 10 forks source link

Add support for using thumb stick as simulated mouse #139

Open duhrer opened 6 months ago

duhrer commented 6 months ago

If we make a transparent overlay and absolutely position a cursor image within it, we should be able to simulate a mouse pointer. We would need an approach to detect the foremost object under the cursor.

With a cursor and an awareness of what's under it, we could simulate mouse clicks in a way similar to what we do with the "button tab navigation", but with a MouseEvent and as many of its properties as we can simulate. I'm not sure if we can also simulate mouse hover, but it's worth investigating.

If at all possible this should also be usable with our modals.

This could also lay the groundwork for #77, although additional work to simulate the look of dragging content would be necessary.

duhrer commented 6 months ago

One concern here is that there are a lot of thumb stick actions and only two thumb sticks.

I was thinking we might address this by having a single "smart" action with an associated mode variable, and have an action to toggle through modes. For example, "smart" thumb stick navigation might scroll by default, and depressing the thumb stick (or using the action menu) might switch to "mouse pointer" mode.

duhrer commented 6 months ago

We could also have auto scrolling when the simulated mouse pointer reaches the edge of the window.

duhrer commented 6 months ago

From reading through this and other threads on stackoverflow, it seems like you can't programatically simulate a hover. You can definitely simulate mouseover, mouseenter, and mouseleave/mouseout events.

One approach in the linked thread was to inspect the current styles looking for any that apply to :hover, and then generate a workalike style that uses a named class instead.

duhrer commented 6 months ago

I assume we'd need a performant way to come up with a short list of things to check, and to me that short list should be based on visibility.

There are pure JS approaches that use getBoundingClientRect on every element and check parents to see if they're displayed. Although I'm reluctant to rely further on jQuery than we can avoid, they do provide a :visible pseudoselector that might be easier to use.

As we'll probably be polling quite often, I was thinking perhaps of caching the list of visible elements and only updating it if the window size or scrolling change. However, that wouldn't catch things that appear / disappear based on other interactions, so we'll probably just have to see how expensive it is to make the list.

duhrer commented 6 months ago

Given that we can't change the hover of whatever we're clicking on as easily, we could also just change the pointer, like from a circle (not hovering) to a caret (hovering, pointing at whatever it is).