Quick Keyboard Access to Things on Screen

[cretz]

• Basically a tool that takes keyboard focus when triggered (likely via some keyboard combo)
  • Maybe it only stays running/active while the last key of the trigger is down and disappears on up
• Initially it's basically headless/transparent and inspects what's on the current screen (e.g. win32 handles, accessibility info, etc) and highlights/outlines areas it can recognize as text or things
• Via typing, can essentially "choose" one of the potentially hundreds of areas, and once chosen can do many things including different forms of clicking or setting focus
• Basically just a fast way to jump around to things on the screen when a specific app itself may not offer it
  • Could even support off-screen with some kind of sublime/vscode-like area listing with intellisense-esque auto complete
• Related in nature to #56

[ryan-williams]

fwiw I use vimium for this in chrome and it has changed my life ("f" to make letters appear over every link, typing a letter follows that link)

i occasionally try to use shortcat on macOS outside chrome, but either it's not as good, I don't know how to use it as well, or macOS apps generally are worse than the web at correct use of accessibility APIs that these things leverage (probably some of all three)

[cretz]

Ah, "shortcat" is basically what I was going for here but a few more features. Thanks for point that product out!

[pmf]

inspects what's on the current screen (e.g. win32 handles, accessibility info, etc) and highlights/outlines areas it can recognize as text or things

Fullscreen OCR + something like vim's easymotion or vimsneak would be the ultimate implementation thereof.

[corbie]

macOS-only, but I’ve used this for over a year to good effect: https://vimacapp.com/

Manual: https://github.com/dexterleng/vimac/blob/master/docs/manual.md

Workflow:

• One global hotkey activates an overlay with keystroke labels (e.g. “K”, “LL”, “FDX”) for every visible clickable element. Type the keys to generate a mouse click and move the cursor there.
• Another hotkey activates a scrolling mode for the focused window. In scroll mode you move the window content with Vi movement keys.