Evaluate Editor UI Frameworks

[ilexp]

While the current editor is in a very good state and WinForms is fine, it does have the major drawback of being Windows only - and Mono can't really fix this in a good way.

This is a long term investigation. Neither do I plan to start implementing something anytime soon, nor do I currently have the resources to do so for a change on this scale - but it can't hurt to do some research.

Required Traits

• Cross-Platform support for major desktop platforms: Windows, Linux, MacOS
• A versatile Docking Control like the DockPanelSuite that is currently used. Really, there is no way around this.
• An OpenGL widget that can be used to render the game somewhere in the editor.
• High-DPI awareness. (See issue #140)

  Desirable Traits

• Written primarily in managed code / C# - if it can be avoided, don't introduce natively compiled things into the otherwise managed Duality world.
• Being able to deploy the same binaries on all of the supported systems with runtime detection what backend to use - similar to the goal of Duality's cross platform support.
• Being able to embed the UI framework itself inside the game, allowing to use the same UI framework both ingame and in the editor - which would make developing editor plugins a lot more straightforward.
• Separate style from structure, providing proper skinning support.
• Separate model from view, hopefully making the overall UI code cleaner.

---

There is nothing wrong with WinForms or the current editor, and while there sure is room for improvement, a UI framework change would be a considerable effort that should not be decided easily. This issue is not about starting something, but about looking around what options there are for the long term future.

[ilexp]

The Perspex framework looks promising, but it is unclear whether some of the requirements are met. I have created an issue to talk about their future plans.

[mlaskus]

Maybe Qt?

It's cross platform, can use OpenGL, can do High-DPI and has a docking widget.

I'm don't think you can deploy the same binaries to multiple platforms but other desirable traits should be possible.

A big drawback would be having to rely on some 3rd party project for C# bindings.

[Xinayder]

There's Eto.Forms https://github.com/picoe/Eto However, I'm not sure if you can use the docking controls with it. Perhaps you could fork the project and adapt it to Eto?

[ilexp]

Maybe Qt?

I really like Qt, but this feels like a bit of an overkill - it's essentially a whole framework, complete with everything down to file system access, strings, collections and all sorts of things. I don't really want to introduce a "second .Net framework" here.

Also, I'd like to avoid unmanaged code. (Added a new trait above, forgot that one :) )

There's Eto.Forms https://github.com/picoe/Eto

This one looks neat. Might be something to keep an eye on.

[ilexp]

Took a look at Eto and it seems like the overall concept is to wrap native UI in an abstract way. Favoring Perspex right now for its custom drawing / custom control logic approach, as this has a higher potential to mesh well with the whole "same UI for ingame and editor" idea.

One drawback of Perspex at this point seems to be a focus on very high-level graphics, requiring each backend to implement a full vector rendering stack, which would be a bit too much for Duality purposes. It also seems a lot more heavy-weight than what Duality would really need. Then again, it seems to provide a lot of great functionality.

Research continues. This isn't decided anytime soon anyway.

[ilexp]

Apparently, this has come up before. Linking to the old issue for historic reasons: #159

[ChristianGreiner]

Maybe Xwt? https://github.com/mono/xwt

[sappho192]

Mono launched SkiaSharp, cross platform 2D Graphics API based on Google's Skia library. You gotta see this. https://github.com/mono/SkiaSharp

[Xinayder]

The framework that @sappho192 suggested looks great (I had a quick, quick look). Maybe it could be the first step towards natively supporting android?

[sappho192]

@RockyTV Maybe yes, but maybe not. SkiaSharp just provides graphics API so we need to construct UI framework.

[ilexp]

SkiaSharp just provides graphics API so we need to construct UI framework.

I'm not really looking for a graphics-only framework though - if we were to create a custom UI, we might as well do it on top of Duality itself. Which I personally don't plan to do for practical reasons. Even re-implementing the editor only is a huge piece of work, adding implementing a custom UI framework before certainly doesn't make it easier!

Maybe it could be the first step towards natively supporting android?

Maybe - but providing additional core backends is a different issue than this one here, and should be discussed elsewhere :) Android is not on my radar right now to be honest. Desktop first!

[xgalaxy]

Hey. Just a thought to maybe mull over. You could go a similar route that Unreal Engine decided to go and just render your UI using the engine itself and develop your own widgets, controls, etc. This would be a lot more work but you would be able to hit all of your required and desirable traits.

The way Unreal Engine did it is by have several layers of UI functionality. There is the Slate api which is your low level UI system and its at this level they implement all of the Unreal Editor UI. Then there is the UMG api which is built on top of the Slate api that includes the UI designer for game ui layout and the game widgets and skinning.

[ilexp]

Hey. Just a thought to maybe mull over. You could go a similar route that Unreal Engine decided to go and just render your UI using the engine itself and develop your own widgets, controls, etc. This would be a lot more work but you would be able to hit all of your required and desirable traits.

Yep, this is what I would consider the ideal solution. I don't see this happening in the foreseeable future though, simply because it would be so much work.

[ilexp]

Spinning the thought a bit more, what would be required for Duality to build a UI system that is sophisticated enough to serve as a basis for the editor UI?

• Multiple Windows.
• More sophisticated window handling API, including event handling.
  • OpenTK may or may not be up to the task, there may be some OpenTK contribution involved.
  • Make sure its SDL2 backend would still work for the usual single-window game application.
• An explicit main loop and message pump for launcher applications.
• Global (rather than window-local) input handling for mouse and keyboard.
• More sophisticated drawing API, presumably based on the existing Canvas.
  • Better transformation / "virtualization" capabilities.
  • First: 9-Patches.
  • Later: Gradients? Paths? Geometric algebra?
• A better text rendering system than the current FormattedText implementation can provide. It needs to be much faster and - potentially - more sophisticated / flexible overall.
• A lot of consideration for UI drawing performance.
  • WinForms has a near-zero overhead while nothing is updated explicitly. Ideally, a replacement would match this as closely as possible.
  • Expect very complex UIs to be drawn and deal with it.
• A layout engine.
• A setup that allows skinning and customization to the degree that it's usable in-game.
• All the basic UI controls in a very polished, usable state.
  • A thought for window management. Maybe adopt a model similar to Qt where you don't have to care and the simple rule is "widgets without a parent are in their own native window".
  • Label, Button, CheckBox, RadioButton, ScrollBar, Slider, TextArea, GroupBox, TabArea, Panel, ComboBox, TreeView, ContextMenu, Tool- and Menustrips
  • I've actually done this before in 2009, but this would need to step it up a bit. Still took a lot of time even back then.
• More complex UI controls which are required by the editor specifically.
  • Most importantly, a DockingControl that is as usable and sophisticated as the current one.
• A system design that supports encapsulation enough for a stable editor even when the game itself throws exceptions and leaves rendering, input or audio in an inconsistent state.
  • Also make sure no bit of code can cross the threshold between game and editor inadvertently.

...and then there's also the whole "porting the editor" set of tasks.

Summarizing this, a custom UI would be a really huge workload. Virtually the only way I would be able to pull this off would be if it was my actual job five days a week and even then things would take a noticeable amount of time as long as I don't have a small team helping me along the way as well.

And yes, it's already Open Source and contributors can help, but that only does so much. For a system of this size to succeed, there needs to be a significant amount of technical leadership, which also consumes a lot of time itself, and it's complicated by the fact that Open Source contributors tend to "come and go" - so a lot of the transferred and taught knowledge doesn't even stay in the project, but needs to be taught again when the next contributors arrive.

So.. yeah. This is probably the long version of my above "I don't see this happening in the foreseeable future".

@xgalaxy Please don't read this post as me trying to prove your idea impractical, but as an acknowledgement of its appeal to me and the project, yet I want to put out there what I'm up against if I was to approach something like this.

[RabbitB]

I don't want to recommend any particular library or offering, but has there been any thought into using an HTML5 renderer? Most of the offerings I can think of are true open source licensed (MIT or BSD), as most are also built upon the Chromium renderer.

It sounds bizarre, but when you really think about it, everything you need comes together. Most of the basic controls are natively supported by the rendering engine; everything is customizable; completely platform agnostic; easy to write in with lots of 3rd party WYSIWYG editors. HTML5 and CSS3 are completely dynamic too, so having a JUICY (do people still use that term?) interface that animates and reacts is fully supported without additional work.

I've personally never used an HTML5 renderer for this purpose, but from what I understand, implementing it for a real-time program breaks down to a handful of fundamental steps all of them share:

• Setup your own real-time rendering context using OpenGL/DirectX/etc.
• Initialize the HTML5 renderer; as far as I know, all libs setup and handle their own rendering context. Doesn't seem to be an issue though; I know for Ogre3D, someone implemented Chromium Embedded Framework for use inside the engine, and they just let it handle its own rendering context.
• Load pages for it to render, local or fetched from the web.
• Pass user input from your rendering context into the HTML5 renderer, so the page can be manipulated (handling roll-over animations, clicking controls, etc.)
• Render the page to a texture, and import that into your own context, to be rendered however you like. From what everyone says, this is the most inefficient part, because you essentially need to copy and render a screen-sized texture twice. You render the HTML5 output to a texture, copy that texture data from the GPU to RAM, then copy it back onto the GPU so your own rendering context can use it, where it can be used as a traditional texture.

For handling a traditional orthographic overlay UI (such as the editor), you create a full-screen quad and just render the texture directly to it, with the source page being 1:1 with the rendered texture. Nothing stopping you however from treating the source page as a sprite-sheet, and displaying it on dynamic elements for in-game interactive UI. At least Chromium based renderers are entirely capable of rendering with transparency, so there's no additional work involved.

[ilexp]

I don't want to recommend any particular library or offering, but has there been any thought into using an HTML5 renderer?

I actually have! Nice writeup of this approach by the way, thanks for adding it. :+1:

While I like the idea of separating structure (HTML), style (CSS) and logic (JS) completely, JavaScript is a terrible language that I don't want to drag into the Duality ecosystem. A also don't want any kind of secondary runtime / VM for this, so there would be the need for engineering a solution where the code part is still handled in C#, or differently put: I'd really emphasize the HTML Renderer part of this, as opposed to considering hosting an actual website in there.

This would of course rule out fetching anything from the web, as well as all the JS frameworks out there or even anything that contains a single line of JavaScript, which would probably rule out most pre-made UI components that are available on the web. It would mean to only use a very limited subset of web technology, basically only reducing it to the HTML structure / CSS style part and maybe even removing some more functionality that doesn't make sense outside the web context.

If you're following along until here, then you might notice some conceptual similarity of this very reduced HTML / CSS / C# logic combo with XAML. Which would put Perspex (now renamed to Avalonia) back into the spotlight.

[xgalaxy]

Hate to resurrect this old ticket from the dead but I thought you'd find this library interesting:

Qml.Net - Qt/Qml integration/support for .NET https://github.com/pauldotknopf/Qml.Net

[michael-hawker]

What about Uno?

[ilexp]

It was just announced that there will be a cross-platform UI solution from Microsoft starting with .NET 6, could be worth a look as well:

• https://devblogs.microsoft.com/dotnet/introducing-net-multi-platform-app-ui/
• https://github.com/dotnet/maui
• Roadmap (Wiki)
• Status (Wiki)

[Barsonax]

Iam watching the The Journey to One .NET. session in microsoft build which should mention the new cross platform UI tonight. Hoping to hear more details.