DirectGUI non-mouse interactability

[fireclawthefox]

Description

Introducing a system that enable DirectGUI to be used with all input methods provided by Panda3D, not only the mouse and (in some cases like text entries) keyboard.

This system should not interfere with existing behaviors and application hence either be implemented for Panda3D 2 or as non-breaking change with the need for it to be explicitly activated by the developer.

Due to the complexity of UIs that can be created with DirectGUI, there has to be made a few considerations while implementing this feature.

1. There should be a simple to use default behavior, probably similar to exiting modern GUI toolkits with tabstop definitions and a way to define what input will trigger UI navigation
2. For more complex UIs navigation should be customizable for example if the navigation doesn't follow a linear but more tree like path
3. If possible, at best this feature has to be implemented in the core PG system to be available for all UI widgets, even if they are custom ones and also from the C++ side of Panda3D

Use Case

Navigating GUIs created within Panda3D with other input methods than the Mouse will help make the engine and games developed by the engine more controller/keyboard friendly and also simplifies developing accessible applications for people unable to use the mouse.

[fireclawthefox]

@Moguri can you please also add the directgui tag?

[ArsThaumaturgis]

This one is potentially quite complex--but also potentially quite useful to have.

In case it's useful, my own approach to this has used a system of "navigation nodes" that store their neighbour-nodes, all contained within a "navigation map", thus allowing the user to move from one node to the next.

It can take some work to set up (as much of the assignment of neighbours is currently done largely-manually in code), but it seems to work well thus far.

[Moguri]

While I agree that C++/PG is the preferred spot to implement this, implementing something as a third-party Python library (i.e. targeting DirectGui) would be a great way to prototype and get feedback on what such a system would look like. It could eventually be upstreamed into Panda, either directly as a DirectGUI feature or ported to C++/PG.

[fireclawthefox]

Right, don't need to put it on the C++ side right away, but if we don't we should consider working preferably with already existing structures and features to ease the transition once we want it to have deeper in the core.

One way I can imagine this to work would be with utilizing the nodepath structure. Since UI elements also live in the scene graph, we could say that by default the flow follows the setup of the node tree and uses the already existing sort order for navigating, skipping not selectable nodes like frames and disabled buttons, etc. This will probably be a depth first walk though the scene graph.

That way an implementation effort in the DirectGui base widget could looking like that:

1. Add a new option defining if the element is selectable in DirectGuiWidget (defaults to False)
2. Implement the interaction methods in the DirectGuiWidget or DirectGuiBase class
   1. Should implement a navigate_next method with a generic direction parameter (could be up, down, forward, backward, whatever)
   2. Should implement an activate method to call the widgets main feature (e.g. button click, give focus to entries, start scrolling on sliders and scrollbars, etc)
   3. Should enable an optional deactivate method to stop a previous activate (probably not for buttons but to stop scrolling or take away the keyboard focus from entries)
   4. Should implement a highlight and un-highlight method to show which widget is currently selected and optionally different view for active/inactive states
3. Add a property defining if this property is currently selected (not exactly sure where this would live and must make sure to no conflicting widgets are selected at once (e.g. open dialog button and OK button of that opened dialog), consider multi-select and UI hierarchy)
4. go through all widgets and set selectable active for all that require it
5. override the active method in selectable widgets to define their special behavior.

[ArsThaumaturgis]

Hmm... I'm not quite seeing how you find the next widget in a given direction. Given a scene-graph of DirectGUI widgets, and given a specific widget being currently selected, how do we determine which widget is the next in, say, the "left" direction?

[fireclawthefox]

Finding widgets can be done through the parent/child structure and the sort order in the tree.

e.g. if we have the following structure:

The first selected element will be "Entry 1" since it has the lowest sort value and is highest selectable in tree. Then for example if the user uses for example Tab to head to the next widget the selection will jump to "Button 1" as that's the next sort value in the children of "Frame 1" which is parent of current widget "Entry 1".

Followed by the button we'll entering "Frame 2" and do the same within there. If we have, for example an element with Sort:4 on "Frame 1", this will be activated when the user wants to navigate to the next element from within Frame 2 Button 2.

The same will also work reverse (i.e. moving backwards from Button 1 to Entry 1) and if desired in vertical movements, e.g. if the up arrow is pressed when focusing Button 2 it will move out of Frame 1 and back to the previously selected widget in that level.

There may be cases that a selection context should not leave a special area, but I think that's better of handled by custom code, overriding the proposed navigate_next method.

[fireclawthefox]

Oh and, if we don't have a sort value or equal sort values, we'll best go by the order the widgets have in the scene graph given by get_children which will most of the time be better than trying to do some positional logic for finding the next best thing.

[ArsThaumaturgis]

But that only gives us two directions: forwards and back. What if we want the user to be able to navigate in other directions?

For example, let's say that we have a grid of nine buttons in our UI. If the user currently has the central button selected, then they might expect to be able to navigate to any of the buttons directly above, below, to the left, or to the right.

I'd suggest rather that widgets be able to specify their neighbours in a given direction, allowing us to jump from one arbitrary widget to another arbitrary widget according to the designer's layour.

[fireclawthefox]

Yes, in that case you'd need to define which widgets are around the selected widget. This may come especially complex if you have, say, a dynamic pie menu or skill tree with plenty of branching. But I wouldn't want the developers or designers to always must create such a map. It should be there if its needed like in those cases but for more simple UIs like a start menu, log in screen or option menus which are quite common this may just become tedious and should work without a dedicated map.

[fireclawthefox]

Also, if such a map would be integrated, it probably must be a 3 dimensional map since DirectGUI can also live in the 3D scene rather than just in 2D space and when someone is creating a fancy 3D scene UI where the user can navigate in all directions, this map should not be the point that is limiting the devs in the creation of such setups.

[ArsThaumaturgis]

Yes, in that case you'd need to define which widgets are around the selected widget. ... But I wouldn't want the developers or designers to always must create such a map. ...

That is fair.

What I'd suggest, then, is that DirectGUI by default construct such a map in the background with the behaviour that you describe, but allow the developer to specify their own map if they so choose.

That keeps the system consistent, while nevertheless allowing for both types of behaviour.

Also, if such a map would be integrated, it probably must be a 3 dimensional map since DirectGUI can also live in the 3D scene rather than just in 2D space ...

A good point.

I wonder whether it would be feasible to have the map be dimension-agnostic--that is, to design the system in such a way that it can be used for 1D, 2D, 3D--and even higher-dimensional--UIs depending simply on how the developer constructs it...

But then, perhaps supporting four- or five- dimensional UIs is a little bit of an overkill... ^^;

No, I think that your suggestion is better!

[fireclawthefox]

What I'd suggest, then, is that DirectGUI by default construct such a map in the background

That's what I thought too, don't want to have different logic for one behavior, just need to fill the map in case it's not set by the UI creators.

I wonder whether it would be feasible to have the map be dimension-agnostic

Going beyond the third dimension is probably hard to do and a waste of resources since it will only be a very rare case in which the default behavior may not work anyway. I mean how many time based (as I guess would be the 4th dimension mostly) UIs are there yet? For such cases overriding the methods for navigation will probably be the best way to go and gives enough flexibility to work with any dimensions.

[ArsThaumaturgis]

That's what I thought too, don't want to have different logic for one behavior, just need to fill the map in case it's not set by the UI creators.

That does sound good then, I think! ^_^

I mean how many time based (as I guess would be the 4th dimension mostly) UIs are there yet?

I was actually imagining four (or more) spatial dimensions, I believe. Tesseract UIs? :P

But no, you are correct I do think in recommending that we stick to a maximum of three spatial dimensions for the standard implementation. Those who do want higher dimensions can, as you say, sub-class and override as called for.

[fireclawthefox]

I was actually imagining four (or more) spatial dimensions, I believe. Tesseract UIs? :P

You see, the first thought I had was UI controlled by intervals moving around and dependent on the time and position of them the user may navigate somewhere else, which could for example be part of a game logic.

So, I'd say the next best thing we can do here is looking for implementation details like what's existing for reusing, which functions are needed where and finally start with some first implementations.

[ArsThaumaturgis]

Hmm... I'm not aware of any such functionality in DirectGUI. (Although I have some vague recollection that it may have some degree of tab-order support?) Thus I think that one would likely be starting pretty much from scratch.

That said, I do stand to be corrected.

[fireclawthefox]

yeah, I was rather talking about the default core features like getting the sort value and scene graph structure to build up the default movement map. Also thinking about which widgets should be selectable by default and how their behavior when selected should be. Especially tricky may be the entry box which catches most keyboard entries like arrow keys and tab and so on, so this may require a default override for jumping out when it has focus.

But I guess most of those questions can be answered with a quick prototype which I could probably just start working on.

[ArsThaumaturgis]

yeah, I was rather talking about the default core features like getting the sort value and scene graph structure to build up the default movement map. Also thinking about which widgets should be selectable by default and how their behavior when selected should be.

That does make sense, I do think!

Especially tricky may be the entry box which catches most keyboard entries like arrow keys and tab and so on, so this may require a default override for jumping out when it has focus.

Indeed, I do recall it proving problematic in my own implementation!

But I guess most of those questions can be answered with a quick prototype which I could probably just start working on.

It does seem worth a shot! ^_^