Feature Request: UI system

[eanders-ms]

Develop a declarative UI framework.

Investigate using AdaptiveCards: https://github.com/microsoft/AdaptiveCards

[sorenhan]

Here are my thoughts on what a solid UI system should offer:

Performance and ease of layout are key - many realtime experiences spent way too many CPU cycles (and too much engineering time) on UIs that look very simple, and often the complexities of UIs are underestimated. So this must be designed with this in mind.

• UI controls and 3d elements often have very different requirements, and we would not want to have an actor per UI control. Instead, I expect us to have a UI panel component represent a whole page layout including a hierarchical set of 2d controls, with clipping.

This UI control should still be a component on an actor, with all the benefits that comes with it: parent transform for 3d placement and animation, parent visibility and collision, exclusive actors, actor groups, and synchronization

Minimizing latency is key, so therefore all the default input functionality needs to be running locally. This includes behaviors for hover highlighting on controls, and control click down/up state visual hints, text entry, scrollbar movement. The actual changes in state when clicking etc. needs to be synchronized to other clients through regular MRE synchronization methods. It should be possible to register behaviors that can listen to feedback from the UI panel, similar to our button/grab behaviors.

scrollbars should optionally be synchronized - it should be up to the MRE author to decide which scrollbars are synchronized and which are not.

Note that each user may see slightly different views, because of

• large font mode for accessibility
• different languages with different text lengths
• right-to-left fonts
• dark mode/various colorblindness color schemes

Rendering optimizations:

• minimal per frame CPU cost, especially when nothing is changing. Reuse geometry buffers, and perform only partial updates.
• minimal number of draw calls for UI control
• Shader-based region clipping and scrolling support, so scrolling a pane around only requires updating a single shader constant rather than rebuilding vertex lists each frame.