Windows Terminal architecture doesn't work in Mixed Reality environment

[PhMajerus]

Testing on Windows 10 1903 (build 18362.86) with a Mixed Reality immersive (VR) headset.

Windows Terminal is listed in the Classic Apps folder in the WinMR start panel (typically for Win32 and legacy WinRT 8.x apps) instead of in the main apps (All Apps) list like modern (UWP apps). When launched in the mixed reality home environment, it doesn't open properly, and instead loads for a while and then shows to the splash image instead (which happens when an app fails to load). Simultaneously, an instance of the shell it is supposed to connect to gets launched in a full windowed Conhost instance on the desktop instead of redirected through ConPTY to a Terminal instance.

I know you might consider Mixed Reality to be far from the command line, but the Windows Terminal could be precisely the opposite, bringing the terminal to devices that have no 2D screen attached, such as HoloLens 2, or later, all-in-one VR headsets. Even for enterprise use, I can clearly imagine scenarios where connecting to a server with a floating augmented reality terminal while handling the hardware could be a great help in the servers room or when working with IoT devices in the field.

Note Conhost can actually be launched as a mixed reality slate from the Classic Apps folder, but the modern Windows Terminal could provide much better rendering, and could be available in the main start panel.

[zadjii-msft]

Honestly, the fact that it opens a conhost at all is probably farther than I thought it'd get in MR :P

We definitely don't have any VR experience on our team, so I can't say we've started on VR support at all, but I like the initiative. It'd be really cool to have the terminal working there.

How, however, is the real question.

We are using UWP UI elements for much of the UI, that's correct, but we are a Win32 application first and foremost. I don't really know much about how either Desktop Bridge applications or XAML Islands interacts with VR, so that might have something to do with it.

if someone wants to investigate getting the Terminal to work right in VR, I certainly won't stop them :)

[BaRRaKudaRain]

Interesting...

--

An outsider, the BaRRaKudaRain

-----Original Message----- From: "Philippe Majerus" notifications@github.com Sent: ‎5/‎11/‎2019 4:56 AM To: "microsoft/Terminal" Terminal@noreply.github.com Cc: "Subscribed" subscribed@noreply.github.com Subject: [microsoft/Terminal] Windows Terminal architecture doesn't work inMixed Reality environment (#693)

Testing on Windows 10 1903 (build 18362.86) with a Mixed Reality immersive (VR) headset. Windows Terminal is listed in the Classic Apps folder in the WinMR start panel (typically for Win32 and legacy WinRT 8.x apps) instead of in the main apps (All Apps) list like modern (UWP apps). When launched in the mixed reality home environment, it doesn't open properly, and instead loads for a while and then shows to the splash image instead (which happens when an app fails to load). Simultaneously, an instance of the shell it is supposed to connect to gets launched in a full windowed Conhost instance on the desktop instead of redirected through ConPTY to a Terminal instance. I know you might consider Mixed Reality to be far from the command line, but the Windows Terminal could be precisely the opposite, bringing the terminal to devices that have no 2D screen attached, such as HoloLens 2, or later, all-in-one VR headsets. Even for enterprise use, I can clearly imagine scenarios where connecting to a server with a floating augmented reality terminal while handling the hardware could be a great help in the servers room.

Note Conhost can actually be launched as a mixed reality slate from the Classic Apps folder, but the modern Windows Terminal could provide much better rendering, and could be available in the main start panel. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[mdtauk]

@zadjii-msft The 3D glassy treatment of the Terminal tab in that Build trailer, is probably why there is expectation of the Windows Terminal in a MR scenario.

Whilst I don't think there is a good enough typing experience in MR for it to be totally practical and useful. Ensuring it works in Universal Windows platforms should be considered, including Windows Core OS, should be on the ToDo list.

[PhMajerus]

19H1 added support for Win32 apps in HoloShell as individual slates instead of through a Desktop view window. However it's still not very reliable, and even Office desktop apps do not launch properly. Hopefully this is good news for the Windows Terminal, if it's just a problem with XAML islands, it will probably be fixed without any work on your side. I guess I should try to write a simple Win32 GDI app that uses cmd.exe through ConPTY and just display the last received line or something, if that works in HoloShell, then the ConPTY architecture is ok and Windows Terminal can get fixed at some point. If that doesn't work, then it's bad news for the ConPTY architecture itself.

@zadjii-msft Win32 apps always appear as second-class citizens in the MR Start panel, and weren't supported at all until 19H1. This could change later when it gets more reliable though, but the focus is definitely on UWP apps. Centennial (Desktop Bridge) doesn't change anything here, Win32 GUI and CUI apps distributed through the Store show up in the same place and run the same. The good news though is that even Win32 CUI apps are supported : (screenshot shows cmd, PowerShell and AXSH running in individual Conhost slates in the Cliff House)

@mdtauk Yeah, the 3D video was cool, but that's not the reason I considered MR. I worked for years with Windows Embedded mobile terminals with keyboard and all kind of peripherals. I believe AR devices like the HoloLens can be an evolution of that market, so controlling RS232, IoT through telnet or shh, etc... from a HoloLens came to mind immediately. You're right that the immersive (consumer VR) headsets are not easy to use with a keyboard if you can't do that blind, and the VR keyboard isn't fast. But on augmented-reality (HoloLens) headsets, you can see your physical keyboard, so a foldable Bluetooth keyboard or a pocket keyboard (such as Rii Tek ones) makes a great typing experience. Configuring a bunch of IoT devices with a HoloLens and just a small keyboard would be much more mobile than having to carry a laptop around. It can even work for legacy equipment with some adapters such as a Bluetooth RS232 dongle for serial terminal support. If you add my #694 suggestion, you can also make the connection very easy and empower IoT technicians.

[driver1998]

Maybe you can try FluentTerminal, which is a UWP app with a Win32 backend to talk with ConPTY/winpty. It might work better.

[WSLUser]

Whilst I don't think there is a good enough typing experience in MR for it to be totally practical and useful.

Grand vision I had was actually using some type of pen/marker made specially for Hololens and writing out anything you wanted into a terminal. So instead of being stuck sitting at a computer, you have the option to stand and use a flat wall/cubicle surface to write instead. Of course a VR keyboard should still be available if a user chooses to use it and simply prefers to stand and type without actually needing any physical items (stands, etc.). Bonus points if you can actually "feel" the keyboard (yeah, I'm a Trekkie who loves the Holodeck and sees the similarities to it and MR).

[WSLUser]

Also probably worth filing a separate issue but instead of win32, why not go UWP/msix? I think that would make more sense and more it more portable too since msix support isn't just Windows 10.

[zadjii-msft]

@WSLUser msix is just a packaging format, right? I believe the app bundle we create is already a msix.

[WSLUser]

I was pretty sure the conversion was win32 - uwp - msix using the desktop bridge. Though there might be some packaging support to take win32 and wrap it in a msix app bundle.

Edit: Ok, doing a different filter type showed the powershell script with it in there. So basically what I mean is get rid of win32 and go full UWP. I believe the use of Xamarin can help ease that effort to some degree.

[PhMajerus]

@WSLUser & @zadjii-msft Yeah, MSIX is only a packaging format, basically a subset of APPX that can be used for easier out-of-Store distribution and portable to non-Windows platforms.

The question could be why use Win32 with XML islands instead of UWP with a plumbing Win32 dll if needed (if ConPTY cannot be used from UWP). a fulltrust UWP should make that possible.

Imagine a frontend app that's completely UWP, with no knowledge of how to connect to anything, it only knows how to handle the eye-candy rendering and inputs, plumbing is the job of "streams providers". Streams providers could be a plug-in architecture, one for serial/Bluetooth terminal, one for Telnet, one for ssh, and one for ConPTY, not to mention easy to add one for the-next-best-shell-protocol. Windows Terminal for Windows_Desktop could be a fulltrust package that includes the ConPTY relying on Win32. However, when installing Windows Terminal on a Windows platform that doesn't support ConPTY or even any Win32 at all, the package for Windows_Holographic and Windows_IoT could contain the same frontend binary, but without the ConPTY "streams provider".

This would make the frontend app completely UWP, and if some platform that doesn't support Win32 at all were to arrive, it would make the Windows Terminal still able to work as a telnet, shh and serial terminal. This also means when the other teams are finally done porting a subset of XAML and DX to other platforms for the real universal Windows platform, the Windows Terminal could run on Mac, Linux, whatever, and connection to these platform-specific local terminal plumbing could just be optional "stream provider" components like the ConPTY one, say UnixPTY.

[WSLUser]

Yes you are spot on with what I was thinking about, especially after quickly refreshing myself on what msix supports. The Windows Terminal app should be platform agnostic at some point. Because why not add another terminal to the list of terminals to use on different platforms?

[zadjii-msft]

I'm not saying that's out of the question, for sure. In fact, it's part of the design of the current Terminal. Everything that's Win32-UI related is contained specifically to the WindowsTerminal project. Theoretically, you could create a UWP application that just instantiated a TerminalApp::App, and that would work perfectly fine as a UWP application, with the caveat that the conpty would be unable to do anything interesting, due to the limitations of UWP.

Right now, we don't really have other TerminalConnection's available, but the idea is that they very well could be pluggable, so you could theoretically have an SSHConnection that just ran ssh instead of Conpty. In our case, the TerminalConnection is essentially the interface for your hypothetical "streams provider". The TermControl doesn't currently take in a TerminalConnection object, but I think it should, rather than constructing one itself.

We're pretty strapped for dev resources at the time, however, so we've been really focused on getting the Terminal working great on Desktop. I'm certain that creating a pure-UWP version is possible, but it would definitely require a bit of finagling that I just haven't had time to get to yet.

[mdtauk]

UWP Xaml supports the Direct X swapchain panel.

But I think it may not be for some time after WinUI 3.0 comes out, and support for apps outside of the Sandbox is baked into Windows and WinRT.

[DHowett-MSFT]

@PhMajerus weird question -- I recently merged a change to how conhosts are spawned. I'm wondering if that impacted this bug at all?

[PhMajerus]

@PhMajerus weird question -- I recently merged a change to how conhosts are spawned. I'm wondering if that impacted this bug at all?

I didn't have the time to compile my own copy for now, just running the one from the Store since it got published, but it seems it didn't get updated since two weeks, version 0.2.1831.0. Do you know if the plan is to have the store build regularly updated and which version incorporates your conhost changes ?

[WSLUser]

Do you know if the plan is to have the store build regularly updated and which version incorporates your conhost changes ?

https://github.com/microsoft/terminal/issues/1944

[PhMajerus]

Good news everyone! It seems that following Win32 apps in Mixed Reality compatibility improvements, several apps that used to fail now work (including Office 365 desktop apps). The Windows Terminal now works in Mixed Reality, tested on Windows 10 1903 build 18362.327: It is still not as good as a UWP app as the launch time is longer and performances degrade with all Win32 apps (probably because of some RDP-app-like GUI redirection), and the MR keyboard isn't shown when it has focus. But at least it can be used with a physical keyboard. Note I don't have a HoloLens 2 to test it on, I don't know if Win32 apps are planned to be supported on HoloLens 2.

Edit: For anyone curious, the Win32 compatibility improvement is part of Windows 10 builds 18362.329 and 18363.329, from KB4512941 : "Improves the user experience and app compatibility so that more Win32 apps will work with Windows Mixed Reality." (Source: https://support.microsoft.com/en-us/help/4512941/windows-10-update-kb4512941 )

[bitcrazed]

THIS IS AWESOME - thanks for sharing @PhMajerus.

I've been wanting Terminal to work on HoloLens since before we created Terminal :)

Sooo excited that it's starting to work. And as we progress towards becoming a real UWP (once platform supports our needs), Terminal will get better and better on Mixed-Reality devices.

Let me see what we can do about making this EVEN cooler in the future ;)

This said, if this issue is now sufficiently resolved, could you please close this issue?