Idea: Order hints based on screen location

[blackketter]

As far as I can tell, the hints shortcuts appear to be generated in the order that they are extracted from the operating system API.

I wonder if it would be possible to order them based on the screen location.

For example, first character might be generated in alphabetical order, so that hints on the menu bar would all start with A and moving down the screen so that hints at the bottom would start with Z. And the second character might be sorted similarly but for the horizontal position.

That way the Apple menu would always be AA and the bottom rightmost item would be ZZ (limited by the list of keys used, of course.)

This might speed up visual parsing of the hints. It would also mean the same hint would often appear in the same place.

[dexterleng]

@blackketter

This might speed up visual parsing of the hints

Could you elaborate on this? For me, I look at a button I want to click, hit the shortcut, see the text on it, then type it out.

[blackketter]

@blackketter

This might speed up visual parsing of the hints

Could you elaborate on this? For me, I look at a button I want to click, hit the shortcut, see the text on it, then type it out.

I think that users would learn some muscle memory that would marginally speed up typing the shortcuts. Certainly the first few items in a menubar would be constant, and for many apps that have fixed button positions this could be true as well.

I also wonder if having the relative positions be consistent across apps would speed up the brain's processing of the hints. (Z being along the bottom means that I can type Z and then read the next character...)

[dexterleng]

Constant hint text for menu bar items is doable.

Assigning hint text to elements in a sorted element position order is not hard, but has a couple of challenges:

Hint text is linearly generated/assigned

If the menu bar has 10 items, and it grows to 11 while the main window remains constant, it will offset the hints in the main window by 1. It can be solved by giving the menu bar buttons a "window" of say 30 hints, but some hint texts will be wasted.

Views are scrollable, lists are dynamic, which can offset the hints

I believe Vimium assigns by coordinates or element hierarchy or something similar.

You can see by scrolling down, the hint text given to elements that were previously visible are now assigned to the elements below them:

Before: "FD"

After: "AA"

[dexterleng]

Some other ideas to solve this visual parsing problem:

Assign hint text by coordinate spaces

The screen can be broken up into many squares, and hint ranges will be assigned to each square. This way, elements in a certain coordinate space e.g. (0,0) to (50, 50) in the screen will be guaranteed e.g. AA -> AD

Assigning hint key sequences that reflect the location on the screen to press

This is a strange idea - if an element is in the top left, we assign it a hint text that consists of characters in the top left of the keyboard (e.g. QW)

Determine hint by other additional information (e.g. text)

1. sort elements by text
2. assign hint text by first/second character (eg Pull Requests -> "PU" or "PR")

This would come at the detriment of performance.

Hint Filtering by Text

Vimium has a Use the link's name and characters for link-hint filtering option that allows you to cut down the number of hints by searching the element text. I've gotten requests to add a search feature, but IMO this is much slower to use even if it solves the visual parsing problem.

[dexterleng]

I'm sure there are many heuristics that can be used to keep hint text static, the Vimium GitHub issues have some discussion on this subject.

I'll have to do some more research and look into how Vimium keeps hints static.

[blackketter]

Fantastic! Thanks for looking at this. I do like your ideas for mapping screen space to consistent shortcuts.

This is definitely an open-ended problem, but the first step of making the menu positions consistent is a good one!

One thing I notice when looking at hinted screens is that the hints tend to be clustered in rows and columns. It would interesting to see if those clusters are easily detectable and have them share keys in the hints.

That's been done with the menubar with your recent change, but could be done more generally by sorting the Y positions of all the hints then looking for same Y positions and assigning them to the same first keys in the hint. Then do the same with the X positions and assign their second hint character to be ordered and consistent in the X direction. (Does this make sense?)

[dexterleng]

sorting the Y positions of all the hints then looking for same Y positions and assigning them to the same first keys in the hint. Then do the same with the X positions and assign their second hint character to be ordered and consistent in the X direction.

My understanding of this idea is this. You have a screen, you break up into clusters either by:

1. Cutting the screen into little squares/rectangles OR
2. From identifying clusters based on the hierarchy of the UI elements. (e.g. elements of a list belong to the same cluster)

Each cluster will be assigned a letter for X and a letter for Y.

Hint texts will look like: <Cluster Letter for X><Cluster Letter for Y><Unique Letter within the Cluster>

Something like this would allow the user to predict the first two hint characters based on the x/y coordinates.

[dexterleng]

A concern I have here is that I think the human eye beyond a certain point (e.g. 10x10) can't tell the exact cube the element belongs to.

Something like 2x2 or 3x3 would be ideal. OR providing visual rulers along the edges of the screen to guide them and train them to predict the hints.

[dexterleng]

The trade-off here is that hints will always have at least 3 characters. I'm not sure if improved visual parsing time + typing 3 characters < visual parsing time + typing 1-2 characters

[blackketter]

My understanding of this idea is this. You have a screen, you break up into clusters either by:

1. Cutting the screen into little squares/rectangles OR
2. From identifying clusters based on the hierarchy of the UI elements. (e.g. elements of a list belong to the same cluster)

Those are good, but I was thinking of something a little different:

Roughly, I think you can sort all the hints by Y. Then group rows by looking for similar Y values. Items with similar Y values would share the first hint key. Identical Y values, (which is common, like a menu bar or window title bar), would always share a key. You can decide how "similar" the Y values need to be based on how many keys are available.

Then do the same with the X dimension and the second key. If there are more hints than can be done with two keys (XY) then expand it to (XYY) or (XXY), depending on how many unique values there are in either direction.

This approach may not always have the same key be on the same absolute section of the screen, but I bet it would generally be close.

[blackketter]

Thinking about this a little more... Here's another, similar approach:

• Look for hints that have identical X values or identical Y values.
• Assign the first character of a shortcut to the largest groups of identical values in either axis.
• Assign the second character of a shortcut based on the relative position of the hints in the other axis.
• Once all the items in groups that share identical X or Y values are set, try assigning keys for decreasingly similar X or Y values.

For the example below, each group of items highlighted in blue would share a first character. Within each group the second and third characters would be assigned in order.

Then the algorithm would find the group in green once it relaxed its similarity requirement and assign a first character. Eventually it would assign the remaining hints to one of the remaining available characters.

[blackketter]

After some extended Vimac use this morning in Safari, I realized that one thing that seems to slow me down (a little) is the fact that after some small amount of screen change hints often get reassigned.

Assigning hints based on coordinates and grouping hints will, I believe, increase the likelihood that hints will remain relatively constant across invocations. That'll be a really good thing, I think.

[blackketter]

I wonder if there should also be an explicit heuristic that attempts to preserve hint sequences across invocation, based on their coordinates. That way a web page, for example, that reloads with some modified content can have the same key sequences for the targets that remain the same, like navigation controls.

[Mehechiger]

I'll definitely support constant hint position across invocation ;)