Feature request: 3D application development support

[vrenken]

Hi,

as one of the tech-guys involved in today's stream of VR/AR applications I've got the feeling that this domain lacks mature application architectural patterns which have been common in mainstream desktop/web/app development. Traditionally 3D (VR and AR) applications are build using game engines (Unity3D, Unreal, UrhoSharp etc..) which all operate according to the Entity Component System paradigm.

Although a lot of splendid 3D applications can be seen on Youtube I personally get the feeling that the current method of 3D application develop would dramatically benefit from concepts like XAML and MVVM (or MVP/MVC for the matter). Especially in today's time where 'enterprise-grade' applications are being requested.

Because of this I've been tracking PortableXaml and AvaloniaUI for a while now (kudo's to everyone involved), and have started tinkering with some basic tests to see how a XAML+MVVM could be formed and molded in a way that makes the two usable for both 2D -and- 3D application development.

To express some of my thinking - In an ideal world:

• A developer would be able to compose 3D hierarchies using XAML and then bind them on ViewModels using either conventional or responsive based binding mechanisms.
• Having MVVM/XAML could ensure that the 'UI-paradigm' specific layer (2D/3D) could be very thin and the reusable/cross-platform layer build according to .NET standard very thick.
• The same technology would facilitate the re-use of 2D user interface components into 3 dimensions - i.e. it should be possible to position a Window somewhere in 3D world coordinates and enrich it with additional 3D aspects (3D styled buttons for example).
• Such technology would/could boost cross-platform (HoloLens, Htc Vive, Oculus, Magic Leap) and cross-toolchain (Unity 3D, Xamarin/UrhoSharp, Unreal), make it accessible to a broader audience of developers. In overall it could be a splendid fit for a project like AvaloniaUI.

To make a long story short my questions are:

1. Do you guys think this might be something worth striving for, or are there plausible reasons for not doing so that I've overlooked?
2. If it would be interesting, would AvaloniaUI be a possible candidate for it?
3. If yes on #2, has anyone got an idea on where to start? I've got the feeling that such endeavor is quite big. Even thinking about making AvaloniaUI able to cope with 3D positions and sizes already explodes in complexity.

Anyways thanks for all feedback/ideas & keep up the great work on AvaloniaUI!

Best regards from Germany,

Peter Vrenken

p.s. I couldn't find a forum on which to ask this question so I've taken the liberty to use a feature request to trigger a dialog. If this isn't the right place to initiate such a dialog then please feel free to direct me to somewhere else.

[grokys]

Hi @vrenken,

Yes that all sounds quite do-able, however I don't think it's currently in the scope of core Avalonia - we've already got enough to work on ;)

If you'd like to start such an initiative however, you'd have full support from me! If there would need to be changes to Avalonia itself we could try to help.

[grokys]

To answer your questions:

1. Yes!
2. I don't see any reason why not
3. I have absolutely no idea ;) I don't think Avalonia itself would need to handle 3D positions/sizes would it?

As an aside: I've used WPF's 3D support and found it to be sorely lacking, which is possibly why I don't think I've ever seen it used, and why it was removed for UWP. I don't think WPF's example is one to follow...

[kekekeks]

@GeorgeS2019 have you tried using the old WPF's Viewport3D with Viewport2DVisual3D? If we'll start adding 3D support to Avalonia core, that most likely be our API model.

The main problem with using Avalonia as a 3D engine would be the fact that you'll have to create the whole 3D engine. Right now the only thing we have is a PR that allows custom OpenGL context creation and a way to use OpenGL texture as a bitmap (there is also a convenience gl control).

[GeorgeS2019]

@kekekeks U jumped too fast to a wrong interpretation of what I have written :-) sorry for not explaining myself clearly :-)

I am seriously thinking of a translation layer that convert XAML (excluding the WPF3D part in phase 1) to 3D UI (under an open source 3D engine like godot) in VR/AR

I need some tips (the light weight version, perhaps starting with some XAML prototype, before Avalonia) just to provide the bare bone XAML to just convert the UI (2D size, color attributes) to corresponding 3D UI. Then the interactions. Later, the MVVM from WPF to mono MVVM.

[kekekeks]

Our XAML compiler can be used as a standalone library, everything Avalonia-specific is implemented via custom AST transformations. If you only need to compose object graphs, it can be used without extra tweaking.

[vrenken]

Hi @kekekeks, that standalone XAML compiler sounds like the spot-on approach. Somehow I missed it during my initial scan in february 2019. Many thanks for sharing!

Thinking somewhat further: With a standardized XAML schema and a (Reactive) MVVM approach such a setup could even be used to standardize cross-3D application development. In other words the XAML code gets translated into a Unity/Godotnet/whatever node hierarchy which gets its behavior powered by ViewModels and Models.

In my humble opinion this has some very serious advantages:

• Developer onboarding becomes far easier as many developers already understand MVVM but not that much really understand the intricacies of Entity-Component-Systems setups that power today's 3D engines.

• Many current 3D activities still 'feel' to happen in the realm of proof-of-concepts and small demonstrators. Not that much 3D applications really reach the level of what I call 'enterprise-grade' application quality. Primary reason is the lack of decent, proven application architectures. Having XAML/MVVM available for 3D could really boost this whole trend.

• Migrating back and forth between .NET powered 3D engines could then become far easier - whereas currently there is an enormous lock-in. (I still cannot imagine that Microsoft will maintain a dependency on Unity 3D for their HoloLens devices for long. They envision that in time MixedReality will become a first-class IT citizen just like desktop/mobile/app and cloud. For each of those they have build their own dedicated tools and don't rely on third-party stuff in their core. My bet is that they are working behind the curtain to prepare the same for their Mixed Reality ecosystem.

• Just a random creative thought: when XAML could be used to create 3D compositions that behave according to VM logic it would mean that they also become reusable across 'user interface paradigms'. In other words, write code and 3D user interface parts ones and use them on HMD's, in desktop applications and on (AR powered and traditional) mobile devices.

Sorry for my rantings - just got enthusiastic about the standalone XAML compiler.

Best regards,

Peter Vrenken

[onehundredfeet]

I am investigating building a runtime compatible with Veldrid. https://github.com/mellinoe/veldrid

[riveranb]

@vrenken Maybe not exactly what you want or talk about. But there is a WPF-like UI tool/engine for game develop. NoesisGUI (https://www.noesisengine.com/docs/Gui.Core.Index.html) It is not free, but it is dedicated for real-time game development with high performance, and it is cross-platform and already had been integrated into several game engines (Unity / Unreal). You may want to take a look on their demos. https://www.noesisengine.com/xamltoy/d17807def7bb2f6ad3aa46f33d1fb49e

[GeorgeS2019]

@vrenken @riveranb The use case for NoesisGUI is valid. We are attempting doing something related based on open source MIT game engine BUT targeting the latest .NET 5 features. e.g. :NETStandard 2.1, SDK style project and Web 3D etc. using XAML

[breynard0]

You could add support for something like Monogame, Gemini or another similar library instead.

[zejji]

I would also love to see a way to display a Monogame buffer in an Avalonia window - this would open up a whole new realm of possibilities!

[sboutin4]

I would also love to see a way to display a Monogame buffer in an Avalonia window - this would open up a whole new realm of possibilities!

I naively assumed this was already possible. I'm in the process of porting my XNA engine and editor to Monogame and Avalonia so it can be multi-platform. Being able to render whatever I'm doing in Monogame inside of Avalonia is a must for me!

After a bit of digging around in the source code, I stumbled upon Avalonia.OpenGL and was under the impression it wouldn't be too hard to port MonoGame.WpfCore using that but maybe it's not that easy?

I'll admit that I didn't look into it too much so this is all based on assumptions. It's not a priority for me right now, the engine takes precedence but if porting MonoGame.WpfCore is already possible then I might give a shot at some point.

[maxkatz6]

See https://github.com/AvaloniaUI/Avalonia/issues/9925 It's now possible to integrate directx and vulkan as well. Samples here https://github.com/AvaloniaUI/Avalonia/tree/master/samples/GpuInterop Available in nightly, also will be part of 11.0-preview5 soon.

Avalonia also supports Rotate3DTransform and possibility to apply third dimensional transformation matrices, but it's more limiting, if you need to have custom low-level rendering inside of the avalonia. See https://github.com/AvaloniaUI/Avalonia/blob/master/samples/RenderDemo/Pages/Transform3DPage.axaml

[JaggerJo]

Stumbled over Monogame integrations for Avalonia:

https://github.com/AvaloniaInside/AvaloniaInside.MonoGame

[GeorgeS2019]

@JaggerJo

Godot4 is a more mature way to bring 3D to AvaloniaUI