Discussing UI options

[oraknabo]

@Aceeri - I wasn't sure about the best place to start a discussion but I wanted to get one going here about different UI options.

---

There are a couple of older ones on the other trackers here and here that cover some of what I'm posting now. This is just a list of different options and some notes:

Engine Integrated

From the short content on the README, I'm assuming this is the direction Citrine is going.

A system that uses Specs and Amethysts's sprite animation, scripting & events/messaging systems whenever possible. Buttons, menus, etc. would just be game entities that emit & recieve messages and listen for input. I imagine this being like Unity's newer UI system (I think the way they use textures for frames on buttons & panels is worth checking out). It could provide a set of core widgets but be powerful enough to do a lot of other things with.

Immediate Mode

While I don't think IM is the best for in-game UI, it can be perfect for editors & debugging tools.

I've been playing around with Dear IMGUI and Conrod lately. While I'm impressed with IMGUI's fairly comprehensive feature set, the 2 or 3 Rust-IMGUI interfaces I've tried aren't as production-ready. Conrod seems decent but seems to still have a long way to go. I haven't used Nuklear but it looks promising.

If IM is better for Amethyst's tools, it might be worth the effort to help make Conrod a more ideal system and just use that instead of reinventing the wheel and focus Amethyst's UI/Citrine system on flashier, in-game styles of GUIs.

Retained Mode

This is the classic desktop-app style of interface like QT, wxWidgets, WPF. I don't think they work very well for games but might be fine for an editor. Sony has a nice looking editor used by Naughty Dog that looks like it uses Windows Form Editor.

Scripted/Multimedia

Slicker AAA-style projects seem to mostly use this kind of system these days. Probably much less efficient than using the integrated and immediate approach, but easier for non-programmers to use since they require the least low-level programming ability.

• Flash-based - highly interactive, heavily animated interfaces often rendered on 3D planes and even complex surfaces. Scaleform is the main example (used in Civ IV, Crysis, Mass Effect, Arkham Asylum, Borderlands, Splinter Cell: Conviction, Deus Ex: HR, Tomb Raider, Skyrim, GTAV)

• HTML/CSS/JS - EA & Maxis have an internal system using a custom-built WYSIWYG editor & parsed by thir own WebKit. As others have mentioned, it would probably be possible to pull WebRender out of servo and use it for something like this. libRocket and Awesomium are similar but much simpler. There are decent arguments for in-game HTML UI but Insomniac recommends against building your tools this way.

Other Options

• Vector-based - This style isn't too popular, but they do exist. Something built out of imported SVGs with a decent scripting language could be potentially as powerful as the flash & HTML versions.

• FRP - I don't have any examples for this, but it seems like it would be possible to do something along the lines of ELM or Reflex that could work in games. Edit: I found this project (by Conrod's creator) trying to recreate some of Elm's functions in Rust--it might be worth looking into.

  I think it might be possible to combine the multimedia approach with the integrated one by adding markup and scripting components to a canvas entity. I don't know if it would be possible to also combine IM to either of those. The concerns for a tools system and in-game GUI are probably too different anyway and it seems best to use a different type for each.

If anyone has anything to add to all that, I'd love to look into other possibilities. I'm also interested in what others would want a GUI system be able to do.

My GUI wish list:

in Editor

• a solid collection of widgets: buttons, sliders, charts, progress bars, scrolling list & text boxes, etc. but also easy to extend with custiom widgets
• the ability to create node editors to manage state machines, animation, effects, dialogue trees and AI scripting visually
• lean API with as little boilerplate as possible

in game

• 3D canvas that can be placed in screen space like a traditional HUD but can also be placed in world space to provide effects like Dead Space's holographic inventory windows or Doom3's screens
• menus and panels easily navigated by controller (directional references stored in each component?)
• easily scripted behaviors, animation & effects
• the ability to make complex, unique interfaces like nested radial menus or unique dialogue option navigators
• the ability to render to a 2D texture from an extra 3D camera in a scene

[oraknabo]

After writing all that, I'm starting to think I'd like to try building a scriptable SVG-based version of Autodesk Scaleform that renders to the GPU.

I found some interesting projects that I think I could learn from to help make that happen: SVG tassellation Gnome LibrSVG NanoSVG SVGL VG in OpenGL GPU rasterizer for vector graphics

Also any JavaScript SVG animation libraries would be worth looking into.

If anyone is interested in making this a standard part of Amethyst or figuring out how to work this idea into Citrine, I'd like to continue discussing it here. I think it's a good alternative to HTML/CSS (which a lot of people seem opposed to here) and could have a tool that basically works like the old Flash interface that lets non-programmers draw components and then write simple scripts to add animation and interactivity.

Actually, I think if the whole engine supported SVG as a sprite format, this could be easily done. Here's something similar for Unity.

[OvermindDL1]

/me is still partial to the DearIMGUI style rendering as it is far more powerful and very efficient, and not hard to rewrite into Rust, though Rust bindings do exist (rewriting it would give a chance to clean out the deprecated parts of its API and so forth)

HTML/CSS/SVG are all not very efficient (no where near), especially if anyone starts talking about having Javascript control it...

[Xaeroxe]

@oraknabo Thank you for your input, I generally agree with a lot of what you have to say. However one thing I do want to make clear right away: Creating our own tools to create art assets is an explicit non-goal for the Amethyst project. There are plenty of existing pieces of software that do these very well and we'd be wasting time to try and replicate them. That being said I do indeed want support for importing SVG files created in other programs in the future.

I'm extremely wary of anything using web technologies (including Flash), as they are typically very inefficient and come with lots of overhead. That being said Scaleform has a rather impressive library of games using it. Unfortunately it's not really an option for us because everything we use must be MIT/Apache2.0 compatible.

[oraknabo]

@OvermindDL1 - I mentoned the performance tradeoff, but I think SVG alone would be much better than HTML/CSS/JS. I think it's worth considering having a UI system that's usable by non-programmers but would be optional for people who think it's a risk to performance. I'm thinking any scripting language used in Amethyst could be used to script it.

@Xaeroxe - If the engine supports SVG, there's probably little need for a dedicated editor. People could build the pieces in something like Inkscape and then import and use the animation system and scripting interface to do most of what I'm talking about.

[OvermindDL1]

I think it's worth considering having a UI system that's usable by non-programmers but would be optional for people who think it's a risk to performance.

That is something that is quite nice about the IMGUI style, it is very simple to create an interface to create the code on the fly and generate out the code for the GUI unlike about any other style. In addition it is simple to fall back to raw code if you want to do very unique effects that any other GUI system just is not capable of. I'm not saying to use IMGUI itself (though bindings do already exist, I think a pure Rust implementation would end up superior) but the 'style' it uses is very powerful and easily generateable.

[oraknabo]

@OvermindDL1 - I definitely think it's good to use the concepts from IM whenever they help improve things. My only concern is that, while IM systems are really powerful for representing and manipulating data in realtime, I've never seen one that can really satisfy the stylistic/aesthetic requirements most games have. That's why multimedia GUIs seem to be popular, even with performance-conscious devs.

Even the creator of DearIMGI says it's intended for debug tools, not in-game UI. Unity originally focused entirely on IM and still use it for the editor inspector, but had to introduce a second more visually extendable system that works better for game UI and requires far less code to use.

I think it would be possible to build something that uses SVG to combine the visual focus and simplicity of the HTML/Flash solutions without having to build in support for either of those technologies inside of the engine framework. Ideally, this would still follow the basics of the Engine-Integrated approach using the animation system and scripting already in the engine.

The simplest version of this would be to import SVG files and use them in basically the same way as other sprites.

The more advanced way is to use similar techniques as JS animation libraries to animate them. SVGs can even store blocks of script inside their source just like HTML pages. When an SVG is imported into a game project, the system could parse groups or layers in it into submeshes and store the visual data as a mesh. The scripts could be extracted and parsed in a way that could be called by other scripts that trigger them based on events, timeline markers, etc.. Scripts could rotate & scale elements and even turn on and off layers based on input and events in the game. It's not too different from the idea of a 3D format storing skeletal animations that get called to run by a state machine in the game.

My attitude about in-game IMGUI may be just from ignorance. If you know of an IM system that looks as good and works as well as some ScaleForm GUIs I would love to check it out.

[OvermindDL1]

I've never seen one that can really satisfy the stylistic/aesthetic requirements most games have.

How so? I've seen them be quite dynamic, simple to apply shader effects to (although the official DearIMGUI's interface leaves a bit to be desired in its shader interfacing there), etc...

but had to introduce a second more visually extendable system that works better for game UI and requires far less code to use.

That mostly comes from it's C heritage. With a decent C++/Rust API it could be substantially shorter in code.

Never heard of ScaleForm so no clue what to compare to it.

[oraknabo]

Never heard of ScaleForm so no clue what to compare to it.

These examples are a little excessive in my opinion but they illustrate what Scaleform can do pretty well.

[OvermindDL1]

Ah 3D stuff, not directly exposed in the IMGUI API in any form (another reason I think it'd be easier to rewrite it) but it would be very easily done with it's style. You could do it quite efficiently (regardless the method) with a rather simple shader actually...

[oraknabo]

I found this interesting:

Is Conrod Immediate or Retained?

Conrod aims to adopt the best of both worlds by providing an immediate mode API over a hidden, retained widget state graph. From the user's perspective conrod widgets tend to feel stateless, as though they only exist for the lifetime of the scope in which they are instantiated. In actuality, each widget's state is cached (or "retained") within the Ui's internal widget graph. This allows for efficient re-use of allocations and the ability to easily defer the drawing of widgets to a stage that is more suited to the user's application. As a result, Conrod should be able to provide the convenience of an immediate mode API alongside performance that approaches that of traditional, retained GUIs.

Also, Conrod's probably moving out of Piston soon.

[OvermindDL1]

... Conrod ...

Ah, that sounds exactly like how Dear IMGUI works, that'd be quite nice then!

[ivnsch]

Any news here? Is there another place where we can follow GUI development progress?