search

mjrusso / scoot

Keyboard-driven MacOS cursor actuator
BSD 3-Clause "New" or "Revised" License
333 stars 14 forks source link

Add element-based navigation mode #1

Closed mjrusso closed 2 years ago

mjrusso commented 2 years ago

This PR introduces "element-based" navigation. 🎉

When this element-based nav mode is active, Scoot will use MacOS accessibility APIs to find the user interface elements on the screen (specifically, those in the focused window of the frontmost app at the time that Scoot is invoked).

Here's what this looks like, in Safari on example.com:

Screen Shot 2022-01-01 at 10 14 11 AM

Scoot identifies the only link on the page ("More information..."), and assigns it the key sequence "aa". (As always, type "aa" to teleport the mouse cursor directly there.) Scoot doesn't just find the content on the web page though -- it also identifies the browser chrome/ UI elements. For example, type "k" to move the cursor to the "show tab overview" button.

Another example, on a web page with (a few) more links:

Screen Shot 2022-01-01 at 10 14 37 AM

Here's Music's MiniPlayer:

Screen Shot 2022-01-01 at 10 24 47 AM
mjrusso commented 2 years ago

Legibility is usually OK-ish, but far from perfect. (Help making this better is appreciated!)

Here's what HN looks like, for example...

Screen Shot 2022-01-01 at 10 30 48 AM

The combination of a small font size, with many unique anchors in close proximity, is problematic. It might make sense to ignore elements that are clustered too closely together (using size as a proxy for importance).

mjrusso commented 2 years ago

Another issue: if the focused window in the frontmost app has a scroll view, and you pull up Scoot (in element-based nav mode), and then start scrolling (either using Scoot's keyboard shortcuts, or using your physical mouse or trackpad), the regions that Scoot has drawn won't update.

In the case where you're scrolling with Scoot, it should be possible to properly handle this. However, the latter might be a problem: are there any accessibility-related hooks to detect if the user is scrolling?