Make keyboard controls a first class citizen

[aksdb]

I hereby want to propose to make keyboard controls a first class citizen in the design of Fyne and especially its widgets.

Some other tickets already talk about accessibility (#1093 and #1285), so I think thinking about inputs is a very important step in that direction. There are multiple disabilities that make it hard to properly navigate a mouse or touchpad, and even with voice control (or joysticks) it is easier to give directional commands (up, down, left, right, next, back) than moving a cursor around.

A widget is already the base for every active component of the application. It can therefore be assumed, that each widget will in some way deal with user input (clicking, scrolling, entering content, etc.).

My proposal would therefore be, to implement the Keyable interface on the base Widget. (Bonus: expose OnKeyDown, OnKeyUp, OnKeyTyped etc. callbacks for custom code outside the widget (for the user of the widget; not for a sub-"class").)

The FocusManager (#1294) can likely deal with the tab order and deal with switching focus.

Examples of key behavior:

• [x] Global: tab focuses the next component, shift+tab the previous
• [x] List: up/down moves the currently selected item.
• [ ] Accordion: up/down moves to the previous/next item, left/right folds/expands the current item
• [ ] Tabs: depending on the orientation left/right or up/down moves between tabs
• [x] Select: up/down changes the current selection
• [ ] Slider: space/enter toggles the state and arrows move to next position
• [x] Table: left/right/up/down navigates between cells
• [x] Tree: up/down changes selected node, left/right expands/folds a node with children
• [x] Check: space/enter toggles the state
• [x] Radio: up/down selects the previous/next item

Special scenario: focus trap

For example a text control with formatting might want to use tab and shift+tab to actually indent or unindent text. For these cases, it must be possible to differentiate between "edit" and "navigate". Giving such a textbox focus, does not enter edit mode. Pressing enter would enter edit mode. Then tab/shift+tab are no longer used for navigation, but are propagated to the widget for use. Hitting escape will leave edit mode and switch back to navigiation mode. The widget is still highlighted/focused (so pressing enter would go back into edit mode, but tab/shift+tab would now navigate between controls).

Therefore it should be possible to mark a widget as FocusTrappable (or something like that). In this case, the FocusManager allows hitting "enter" to enter said trap. Afterwards tab and shift+tab are propagated to this widget (while it has focus). Esc(ape) however is handled by the FocusManager to take that exclusive/trap mode away again.

Focusing a trappable widget via mouse or finger tap whould immediately enter trapped mode.

Why do this "soon"?

The more widgets Fyne has, the more will have to be reworked to get a proper input flow. Given the current recommendation on how to extend widgets (implementing input interfaces yourself) would likely break when implementing those same interfaces on the base widget already. So the sooner this functionality moves to the core, the less trouble will be encountered later.

Also it's easier to design a widget with keyboard control in mind, than reworking existing widgets to be keyboard controlled.

This is also why I propose to implement the Keyable interface on Widget, so every widget implementation knows it has to deal with this. During design.

[andydotxyz]

I agree with the general idea behind this ticket, we should have better keyboard support - and we are working on it. I have updated your list of items to be check boxes, to show which are already complete (as far as I am aware). Select is almost done as well, but needs some further work on menus to be complete.

implement the Keyable interface on the base Widget

I don't think this is the right approach, not every widget will provide key controls and adding to the list of features that someome will have to implement just to get a widget working will get in the way of the ease of development that we are aiming for.

Bonus: expose OnKeyDown, OnKeyUp, OnKeyTyped etc. callbacks for custom code

I'm not sure why this is desirable? Making it easy for developers to override the standard keyboard controls seems counter to the desired outcome of this ticket.

Giving such a textbox focus, does not enter edit mode. Pressing enter would enter edit mode.

I think this sounds counter-intuitive... Most toolkits I have experienced make an input editable as soon as you tab into it. Hitting any key combination to make it editable would be cognitive overhead and a behaviour we would have to teach. Slightly better than this would be to have a shortcut to exit the "trap" widget - I think macOS for example uses Cmd-Tab.

Also it's easier to design a widget with keyboard control in mind, than reworking existing widgets to be keyboard controlled.

It seems that this statement is probably true, but I don't think it's sufficient to break every existing widget to force keyboard controls to be added. Is adding the overhead to every new widget the right thing to do when we are trying to make it easy to get started with?

[aksdb]

I will answer in a slightly different order since I guess one answer might benefit the next :smile:

Bonus: expose OnKeyDown, OnKeyUp, OnKeyTyped etc. callbacks for custom code

I'm not sure why this is desirable? Making it easy for developers to override the standard keyboard controls seems counter to the desired outcome of this ticket.

I was thinking of something like this (kinda-pseudo-code, since I don't have my IDE at hand):

myEntry := widget.NewEntry()
myEntry.SetOnKeyTyped(func(event *fyne.KeyEvent) bool {
  if event.KeyName == fyne.KeyA {
    // do something special
    return true // but let the widget continue handling it
  } else if event.KeyName == fyne.KeyEscape {
    // do something else
    return false // stop processing the event higher up; we handled it fully
  }
  return true // by default, let the widget continue
})

This would basically allow intercepting events or handling them without having to "override" anything. If the event handler is not present at all, that's also fine.

Alternatively it could be designed to work like most middleware http handler for Go's http handlerfuncs are designed, which pass the "next func(...)" as parameter so the handler can decide if it should call the next in the chain or not. That would also open up a nice way to stack / register multiple handlers/listeners.

implement the Keyable interface on the base Widget

I don't think this is the right approach, not every widget will provide key controls and adding to the list of features that someome will have to implement just to get a widget working will get in the way of the ease of development that we are aiming for.

The base widget could always implement some handler. Even if it does nothing. It would probably make sense to propagate the event further then. If the entry doesn't handle it, maybe it's parent widget is interested. If that isn't ... maybe the window is interested. That way an event would always hit the focused component and can be passed down to the lowest level control, but could also be intercepted at any time.

With the aforementioned approach to allow "stop" processing or "continue" processing, this opens many flexible ways to write interactive widgets and reusing existing widgets. IMHO easier than the current approach.

Also: at least in regards to focus, at least every widget will have to be focusable. Even a label, since otherwise a screenreader wouldn't be able to tell where you are and what you want to read. But anyway, it might also be an option to separate between Widget (any complex rendered thing) and Control (or InteractiveWidget) that also implements/offers key/tap/input/etc. handling. A label would be Widget, an entry would be a Control or InteractiveWidget.

Giving such a textbox focus, does not enter edit mode. Pressing enter would enter edit mode.

I think this sounds counter-intuitive... Most toolkits I have experienced make an input editable as soon as you tab into it. Hitting any key combination to make it editable would be cognitive overhead and a behaviour we would have to teach. Slightly better than this would be to have a shortcut to exit the "trap" widget - I think macOS for example uses Cmd-Tab.

For normal entries (simple text inputs, drop down boxes, etc.) yes. For complex controls (think the richtext area of Word) this differs between applications. But true, I don't think there's a "standard".

[andydotxyz]

This would basically allow intercepting events or handling them without having to "override" anything. If the event handler is not present at all, that's also fine.

Our design is to keep things simple whilst making it possible to do more advanced things like you illustrate. The code above is entirely possible by extending the widget and overriding - given that changing keyboard behaviour of widgets is not common on most I don't think we should change the design to accomodate the less frequent case.

The base widget could always implement some handler. Even if it does nothing.

If we take the approach of providing empty handlers as an extension of the API I don't think it makes any difference to the current situation. Ignoring the empty methods is just as easy as ignoring the interfaces available for handling these features.

With the aforementioned approach to allow "stop" processing or "continue" processing, this opens many flexible ways to write interactive widgets and reusing existing widgets. IMHO easier than the current approach.

Eactly this behaviour is possible when overriding - developers either call myEntry.Entry.Tapped() or don't from within their own Tapped() method.

[andydotxyz]

I hope that I have responded to your questions and suggestions. I think that this ticket is covering many things in one, and I'm not too sure if I have understood it all. In summary:

• We should have better control for the standard widgets - Agreed and we are working on it
• Breaking Widget API to require keyboard handling will force widget developers to implement keyboard control - I don't think this change is worth the pain for all downstream app developers
• Changing how keyboard controls are exposed to make it easier for developers to customise keyboard handling - The suggestions are all possible through the current API and the alternative approach suggested makes for a more complex standard API for each widget and our design is to provide such features through extension rather than through customisation so that developers are encouraged to keep widgets standard as much as possible.

[aksdb]

Thank you very much for your explanations! I am absolutely fine with this, and of course its up to if you want to keep this ticket around or not. I wanted it more as an entrypoint for discussion and maybe to orchestrate the direction of keyboard support (kind of like an Epic). If it doesn't serve this purpose or doesn't fit the desired ticket style, you can just close it :smile: (or rephrase it, or whatever :smiley: )

[andydotxyz]

I think the ticket is great - the discussion is valuble and should be kept visible. I'll leave this open as the checklist at the top is really useful too (though the details may vary as we progress).

[AlbinoGeek]

Related:

1499 - prevent Keyboard focus on disabled List

885 - Keyboard support for buttons

[AlbinoGeek]

Consider:

• [ ] Form -> (Single Line) Entry Enter calls the Confirm button.
• [ ] Dialog Enter calls the Confirm button (default action)
• [ ] Dialog Escape calls the Dismiss button (cancel action)
• [x] Button (when focused) Space "Clicks" the button. (used with global tab navigation)
• [x] Hyperlink (when focused) Space "Clicks" the link. (tab navigation)

[stuartmscott]

• [ ] Form -> (Single Line) Entry Enter calls the Confirm button.

Yes, if this is the last Entry in the Form, otherwise shouldn't it move to the next Entry as if it were a Tab?

[stuartmscott]

Also, not sure if this is covered by;

Select is almost done as well, but needs some further work on menus to be complete.

But I'd add;

• [x] Select Escape dismisses popup
• [x] Menu Escape dismisses popup

[Jacalz]

I think it would be really cool if the slider could be focusable (like widget.RadioGroup) and the if it could be controlled between steps using the arrow keys on the keyboard. That would be a big step up in my opinion.

[matwachich]

Another idea to discuss, coming from WinAPI, in Select (ComboBox :p), pressing character key selects the item starting with that caracter ; pressing again the same caracter will cycle through all items starting with that caracter.

Another on: in dialog boxes, select the option on the buttons by pressing the first caracter of the text (Y for Yes, N for No, C for Cancel...)

[andydotxyz]

I love the select idea. The "first letter of button" less so - we should support proper shortcut for buttons instead I think...

[AlbinoGeek]

Worth noting @andydotxyz , what they described is how combo menus work in most desktop operating systems and web browsers currently.

It's dual function however.

Consider the following scenario:

Say you have a country dropdown

If I type Canada, both gnome and windows will automatically select Canada.

Naive implementations will end up selecting Argentina or some other country starting with the letter A, because they only implemented the feature the above person mentioned, without also implementing natural or literal matching.

If I type C, it will generally select Cambodia or some other country. I can then keeps tapping C, but I have to wait about a third of a second on gnome, or half a second on windows, and then it starts to go through the countries that start with the letter C, just like they described.

Natural search is much more straightforward, letter search causes issues if the list itself does not have alphabetical order.

However, modern operating systems implement both for native applications.

** When I refer to native on gnome, I'm talking about GTK, yes I know this is a generalization, but it was only meant to serve as an example, not as a rule.

It might be worth considering how existing frameworks handle this feature, because users are trained to expect it to work that way here.

Example:

I have no idea how these handle this by default:

• QT on Linux
• Mac OS X

Apologize for crap layout and formatting, this was from mobile.

[andydotxyz]

@AlbinoGeek I was agreeing with the select behaviour - drop downs etc should have a natural search as you describe.

What I was not on board with was selecting buttons based on their first letter in a dialog.

[psydvl]

What about text highlight with Ctrl? Like Ctrl+← should highlight word before cursor

[matwachich]

Also, another important aspect of Tab navigation is the order of the widgets.

I imagine 2 ways:

• Add a method to Tabbable GetTabNext() fyne.Tabbable
• Add a field TabOrder int that will be used by the toolkit to set tab order

[andydotxyz]

What about text highlight with Ctrl? Like Ctrl+← should highlight word before cursor

Isn't text highlighting done with Shift? we have that in Entry already

[Jacalz]

Highlighting using Shift + Control is actually very useful. I'd personally also vote for the inclusion of that. It is very useful to more quickly select a whole word

[andydotxyz]

Just linking in #3858, though I think it was captured above under "Tabs".

[matwachich]

Ctrl+Enter in multiline entry = Submit ?

[Jacalz]

Shift + enter is already submit in multiline entry

[matwachich]

Also, another important aspect of Tab navigation is the order of the widgets.

I imagine 2 ways:

• Add a method to Tabbable GetTabNext() fyne.Tabbable
• Add a field TabOrder int that will be used by the toolkit to set tab order

I have a better idea (after gaining more understanding of how fyne is working):

Just add a method to Focusable: NextFocus() fyne.Focusable

If implemented, this function will control what's the next widget to be focused, if not implemented or if returns parent.NextFocus(), then fall back to default behaviour.

I feel like this is a very important functionality, worth modifying a standard interface because in complexe layouts (especially when using border layout), the tab order is totally messy and non-logic at all!

---

Edit: we could also need a PreviousFocus() method

[andydotxyz]

A good idea, though let's not make it part of Focusable or it's implementation won't be optional

[andydotxyz]

Collections all completed as of landing #4100