[Remote rendering problem] Local pose reprojection mode doesn't work with Window Mixed Reality XR Plugin

[WikkidEdd]

We were just testing out the new local pose reprojection mode and couldn't get it to work within our app. After a bit of investigation using the Quick Start sample app it appears that the feature only works when using OpenXR.

Unfortunately, we can't use OpenXR in our app as it has some bugs which would block us from releasing.

Is this something that can be made to work on WMR?

Also, I noticed that in your sample you guard setting that parameter on graphics.Api != GraphicsApiType.SimD3D11. Does that imply that this feature is not supported for standalone use? It would actually be really helpful for us if it worked in standalone builds too. We have a desktop viewer application that has a locally rendered environment. When connected to ARR, panning around results in the trademark cropping of the frame of the background environment where it doesn't have enough information to reproject the remote pose correctly

[FlorianBorn71]

Thanks for pointing that out! We are looking into the WMR version now as there seems to be a bug indeed. We are able to reproduce, which is good ;-) Enabling it in Sim/Desktop mode has currently no effect by design. We are looking into possibilities though for better pose prediction and reprojection in desktop mode, but I don't have an ETA yet. Note that reprojection is done by specific HL hardware which is not there on desktop.

May I ask which feature gaps are keeping you from migrating to OpenXr in the first place? Is it something on ARR or dependencies?

[WikkidEdd]

Glad you're not intentionally leaving WMR behind :)

I appreciate it is hardware LSR on HL, but on desktop, you're also doing a reprojection correct? From messing around in editor, the behaviour is essentially doing what remote pose mode does. Rendering the local content at the remote pose, compositing and then reprojecting to current pose.

Can you not just swap that around so it reprojects the remote frame to local camera pose then render the local content? I know it's not quite the same as what local mode does on HL, but you end up with the result. Current pose local content and reprojected remote content.

There are no feature gaps, just bugs/instability. These are the main 2 issues: https://github.com/microsoft/OpenXR-Unity-MixedReality-Samples/issues/52 https://github.com/microsoft/OpenXR-Unity-MixedReality-Samples/issues/53

We've got MS Support Tickets open on these too.

[FlorianBorn71]

What you describe is essentially what local pose mode is doing, i.e. it projects remote image into local pose, renders local content in local pose space (as with any other VR app) on top of it and then does the final reprojection. While this would be a simple approach that we could leverage on desktop as well, we are also looking into improving the quality further. The good thing about desktop is that more sophisticated/expensive projection steps can be involved, because of the much more powerful GPU.

[WikkidEdd]

More sophisticated and expensive would be nice in the future, but the simple and cheap approach would already be an improvement now. :)

Just as you make the local pose mode optional on HL due to performance impact, having the option to do a cheap "local pose mode" on desktop would be beneficial too. After all part of the point of ARR on desktop is that you don't necessarily have a super powerful GPU on your desktop :)

[FlorianBorn71]

yep, we are looking into multiple options and if something simple falls out of it, we'll release it asap and leave the more sophisticated options for later. Unfortunately even the simple approach is not something that can just be enabled right away. But good to know this is a priority for you, so we'll try to consider this.

[FlorianBorn71]

The original bug (local pose in WMR does not work) has been fixed with 1.0.39. And btw, we are looking into local pose mode on desktop as well, so stay tuned for updates ;-)

[WikkidEdd]

Awesome thanks :)

[WikkidEdd]

Local pose mode on desktop makes a huge difference, thanks 🙌