Discussion: WinUI, Mixed Reality and the Future of Windowing

[niels9001]

Discussion: WinUI, Mixed Reality and the Future of Windowing

One of the great things about WinUI, are wide range of input modalities such as mouse, keyboard, touch, controller, voice and even gaze! Another part is the cross platform model that is able to run our experiences on a wide range of device types, and even work in the Mixed Reality world on Immersive Headset and HoloLens.

However, these apps cannot be optimized for e.g. HoloLens UI-wise: it will render the app as a 2D slate. To take advantage of all the unique features of HoloLens developers basically need to start over and built their apps in Unity (although C#, it's basically a complete re-write).

I was wondering, especially since Microsoft is investing so much in Holographic computing, what the long term plan is for WinUI in this space? It would be great if we can some how tailor our experiences to the 3D world with AppWindow and 3D UI styling.

Very much related to that, is the future of windowing. Would be great if the team could shed some light on what we can expect!

At Build 2018, there were some excellent talks about this topic and how teams in Microsoft are thinking about this next gen platform in terms of user experiences. I'm really excited about this and the possibilities it can bring and I'm wondering how the team, and the community is thinking about this going forward. For LOB apps, re-writing the UI in Unity to take advantage of Holographic computing is too much effort and requires additional skills - if it can seamlessly move and be optimized for that with WinUI x.x. / Windows vNext that would be awesome :)!

Related Links

Our Fluent Path to Spatial Computing: Easy as 1-2D-3D (Build 2018) by Mike Harnisch

Building powerful desktop and MR applications with new Windowing APIs (Build 2018) by Roberth Karman

[mdtauk]

Part of this will probably require OS APIs and changes, and not something that can be done entirely within WinUI - but it will be good to see some movement on this, especially for enabling Win32 apps to replace existing UI to XAML, with modal windows, tool windows, and possibly MDI parent windows, with the TabView.

[petroemil]

I think in the context of WinUI what we can realistically talk about is a 2.5D kind of UI elements... with a bit of thickness, some dropshadow, a bit of perspective... maybe positioning windows in the 3D space. But at the end of the day we have to keep in mind the "cross-platform" or "cross-device" nature of WinUI. Something truly 3D wouldn't work on any other device other than Mixed Reality devices.

With that said, I'm looking forward to those 2.5D additions, to give windows and UI elements a physical feel and also to support MR interactions in a natural, intuitive way.

GIFs are from Intro to Building Apps for HoloLens 2 Using Unity and Mixed Reality Toolkit

[mdtauk]

I think in the context of WinUI what we can realistically talk about is a 2.5D kind of UI elements... with a bit of thickness, some dropshadow, a bit of perspective... maybe positioning windows in the 3D space. But at the end of the day we have to keep in mind the "cross-platform" or "cross-device" nature of WinUI. Something truly 3D wouldn't work on any other device other than Mixed Reality devices.

With that said, I'm looking forward to those 2.5D additions, to give windows and UI elements a physical feel and also to support MR interactions in a natural, intuitive way.

There were some early experiments with 3D Markup being explored. Also Build mentioned taking XAML elements, and rendering the controls outside of the window, attaching them to your hands, etc.

[niels9001]

Agreed.. for full-blown 3D experiences Unity would be the better option. But to have 2.5D slates out side of the main view, with 2.5D styling would already be a super great start.

[github-actions[bot]]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 5 days.