search

mjrusso / scoot

Keyboard-driven MacOS cursor actuator
BSD 3-Clause "New" or "Revised" License
333 stars 14 forks source link

Find all elements on the screen, not just those in the focused window of the frontmost app #14

Open mjrusso opened 2 years ago

mjrusso commented 2 years ago

In element-based nav mode (first introduced in #1), Scoot will find all elements in the focused window of the frontmost app.

It would be better if Scoot could find all elements that are on all attached screen(s), across all apps.

Of course, windows can overlap, and we'll need to ignore any elements that are fully occluded by a window above it.

AXUIElementCopyElementAtPosition will find an element at the specific coordinate (if one exists), but you need to pass it an application reference, so it is not directly usable in this context (and, regardless, checking every possible x,y value on every screen seems wasteful from a performance perspective).

https://stackoverflow.com/a/56941105/ has some code that might be helpful (finding all windows on the screen, and ordering them relative to each other).

mjrusso commented 2 years ago

It would be better if Scoot could find all elements that are on all attached screen(s), across all apps.

A stepping stone could be finding all elements in the window that is under the mouse cursor (if it differs from the focused window of the frontmost app). Of course, there's still occlusion issues to deal with.