Architecture suitability for pure GUI applications

[Gerharddc]

I want to spend some of my free time trying to develop a modern Rust-based GUI solution (I'm particularly interested in embedded Linux applications but I would love to develop something that works everywhere) similar to Qt Quick. Even though Bevy aims to be a game engine, it does seem that the UI part of Bevy could work very well for non-game GUIs as well if developed further. In my opinion, the graph architecture and existing 2D rendering capabilities should make it possible to implement dynamic GUIs with smooth animations and even make it possible to seamlessly add 3D elements to the GUI as well.

Before getting involved in the project, I would thus like to get an opinion on wether or not the architecture of Bevy is fundamentally suited towards such an application. In other words, are there lots of design decisions or parts of the engine that are specifically aimed at games which would make GUI performance less than ideal or add a lot of "dead weight"?

[cart]

I think there is nothing architecturally stopping us from building quality non-game UI solutions in Bevy. This is a personal goal of mine as well and we need such a solution for the Bevy Editor. Traditionally, I think the biggest differences between game UI and app ui are "incremental rendering" and power consumption (which are related concepts). I think we can solve both of those problems.

Bevy is super modular, so "ui apps" won't need to pull in other dependencies like 3d renderers or gltf loaders.

That being said, it sounds like the things you want are similar to what we already want for Bevy. Experimentation in any direction is always welcome, but it might be a good idea to get synced up so we can decide if our requirements are compatible. Editor-ready UI is one of our current "focus areas" (#254). The current implementation is already usable for simple things, but I think some aspects (like dataflow) need more experimentation / design work before we start moving forward on the editor. I'm personally planning on diving into that in very near future (as soon as I wrap up some renderer performance work and ECS scheduling stuff).

Could you elaborate a bit on what you'd like your UI solution to look like?

[Gerharddc]

Great to hear that our goals might be aligned. As I alluded to, I ultimately want something that is very similar to Qt Quick/QML, at least in terms of functionality.

So the first requirement is that it should be possible to do the layout part of the GUI in a declarative way, for instance in a JSON type format format that describes hierarchies of components where components can be either high-level elements like buttons or low level-primitives such a rectangles. High-level elements are of course then built using primitives whereas primitives add geometry that should actually be added to the scene graph. This could be 2D or 3D geometry because maybe the user wants to have a slider that rotates a cube.

I specifically want this in something like a JSON file so that one can easily build with a visual GUI designer. On the other hand I also want type safety against the GUI elements, but that isn't particularly important and can be sorted out much later.

All components should then have properties like opacity, scale, width etc. and of course any transformation applied to a component also applies to its child components. I imagine this is how the "incremental rendering" that you mention can be achieved. In other words it might be possible to "cache" the rendering of components into textures rather than actually having them fully rendered for each screen refresh. This means that a button for instance is only re-rendered when its internal appearance changes. If the button simply has to be moved around then this amounts to moving around a square with a texture. This should thus allow for very fast animations on properties that do not require full re-rendering. I would also like to support 3D transformations on 2D elements as well, for instance a button flipping around or whatever.

Something very powerful that Qt Quick (QML) allows is for property bindings. In other words one can specify that one rectangle is positioned 20px to the right of another rectangle. Or you can enforce that one component have 2x the opacity of another etc. Qt achieves this kind of behaviour through a signal slot mechanism and I still need to figure out how I want to do this in Rust. The key requirement is that it should be possible to trigger a re-render of a component in response to property changes. In other words, reconstruct the relevant scene graph elements and re-render the component texture. Alternatively, property changes that do not require re-rendering should inform the higher level component that the texture of its child should now be transformed in a different way.

I want to make it easy to specify that property changes should be animated, but again this can be easily achieved by having a timer internally trigger property changes in smaller increments so the only difficult requirements is to have an efficient rendering response to property changes.

The framework then obviously also needs to deal with propagating input events such as clicks or touches through the components hierarchy. This of course requires that the system have some comprehension of the perceptive field of each control etc. but I imagine this is something that Bevy UI needs in any case.

I also want to make it possible to present data in an efficient way, possibly through a model view paradigm, but I imagine this does not really come with any rendering implications and will lean on the aforementioned property binding system.

A challenge that does pop up with binding data like that is that it might happen that you have a list with say 10 000 items in and the user can at most view 10 at a time so the system needs to implement some form of list virtualisation to only have say 50 components actually instantiated in memory and to then dynamically update which components truly exists while scrolling. I imagine that a game engine needs something along these lines as well to efficiently handle huge maps of which only a small fraction is possible at a time, but the difference in this case is that the components that need to be created are actually the same as those that already exist, but with different data so a further optimisation can be made by recycling the existing components. I am therefore not sure if there is a way to share UI and game code in this regard.

Lastly, the UI system should of course have a reasonable collection of standard components such as textboxes and buttons. These be easy to style, but I do not yet have an opinion on how that should work.