Confusing Documentation on Hololens 2 Spatial Awareness when using OpenXR

[A-Ivan]

Describe the issue

It is unclear if the current Spatial Awareness documentation is targeted at Unity 2020 + OpenXR, Unity 2019/2020 + Windows XR Plugin, or Legacy WSA. In other parts of the Hololens 2 documentation this is clear, as you have the option to choose which is your target (for instance the Holographic Remoting document).

I'm using MRTK version 2.7.3 Unity version 2020.3.24f1 with OpenXR and Hololens 2.

Feature area

I was following along with the "Spatial awareness getting started" documentation and a tutorial called "Made with MRTK: Spatial Awareness" from Microsoft's Developer YouTube channel (published in March of 2021). I know the tutorial uses Windows Mixed Reality as the XR setting and not OpenXR, but I went along with it anyway as I was not able to find any specific tutorial for using Spatial Awareness using OpenXR.

The issue is that when adding a spatial observer to a cloned DefaultHololens2ConfigurationProfile, only the "XR SDK Windows Mixed Reality Spatial Mesh Observer" appears (see image below).

In my case, when using OpenXR and adding a spatial observer, 3 observers appear - "XR SDK Windows Mixed Reality Spatial Mesh Observer", "XR SDK Spatial Mesh Observer", and "OpenXR Spatial Mesh Observer".

First off, I imagine that not all of these observers are needed. Would removing some of them would decrease the amount of resources usage, if so, which ones can/should be removed when using OpenXR? I would imagine the first two because of their names, but this would be good to have documented.

Second, the code (below for easy reference) for showing and hiding the spatial mesh, shown in the documentation, does not work (it does not disable the visibility of the spatial mesh).

// Get the first Mesh Observer available, generally we have only one registered
var observer = CoreServices.GetSpatialAwarenessSystemDataProvider<IMixedRealitySpatialAwarenessMeshObserver>();

// Set to not visible
observer.DisplayOption = SpatialAwarenessMeshDisplayOptions.None;

// Set to visible and the Occlusion material
observer.DisplayOption = SpatialAwarenessMeshDisplayOptions.Occlusion;

I believe this is because it is getting the first mesh observer available, probably the "XR SDK Windows Mixed Reality Spatial Mesh Observer" as it is the first from the Spatial Awareness observer list, and this is not the observer being using when using OpenXR.

From the documentation it is not clear which mesh observer should be used in the case of OpenXR and if more than one observer is being used, how to access other observers (not just the first one).

Is there an updated documentation on using Spatial Awareness with OpenXR?

Thank you.

[A-Ivan]

I figured out part of the solution.

When using OpenXR, if I remove the first two spatial observers ("XR SDK Windows Mixed Reality Spatial Mesh Observer" and "XR SDK Spatial Mesh Observer") and then run the code, to set the spatial awareness mesh display to not visible, it works (the mesh becomes invisible). The problem was that the code was not accessing the OpenXR Spatial Mesh Observer, which is the one actually being used.

This is useful as quick fix, but it would be great if when creating the spatial awareness observer that the system could detect that OpenXR is being used and only create that observer, or at least put the OpenXR observer before the other ones to avoid this issue. A note in the documentation would help.

What still isn't clear though is when using multiple spatial observers, via code, how to choose which one you want to access, not just the first one, either by order or by name.

I am getting the following though error during playmode, but everything seems to be working ok as the scene is able to enter playmode without any issues.

" SerializedObjectNotCreatableException: Object at index 0 is null UnityEditor.Editor.CreateSerializedObject () (at :0) UnityEditor.Editor.GetSerializedObjectInternal () (at :0) UnityEditor.Editor.get_serializedObject () (at :0) Microsoft.MixedReality.Toolkit.SpatialAwareness.Editor.MixedRealitySpatialAwarenessSystemProfileInspector.GetDataProviderConfigurationList () (at Library/PackageCache/com.microsoft.mixedreality.toolkit.foundation@396ac65d719d-1639692003058/Core/Inspectors/Profiles/MixedRealitySpatialAwarenessSystemProfileInspector.cs:58) Microsoft.MixedReality.Toolkit.Editor.BaseDataProviderServiceInspector.OnEnable () (at Library/PackageCache/com.microsoft.mixedreality.toolkit.foundation@396ac65d719d-1639692003058/Core/Inspectors/Profiles/DataProviderAccessServiceInspector.cs:72) "

[wallacewangchao]

I meet the similar issue when using OpenXR with the spatial awareness. Even with the MRTK examples - Demos - SpatialAwareness , the Spatial Awareness does not work. There is no mesh scanned. The Profile is as follows:

I found this issue has been report in #9619 by MRTK 2.6.1, but I am using MRTK 2.7.3 + OpenXR + Unity 2020.3 with HoloLens2, still not working

[mcoy001]

I am using MRTK 2.7.3 + OpenXR + Unity 2020.3 and 2021.2 I also have the same issue. I found the code starts working after I leave only "OpenXR Spatial Mesh Observer" in the settings, and I need to deploy to my device. (Maybe in the new MRTK version it needs to specify which observer is being access to? are we missing a document for that, or it's a bug?) Seems spatial mesh won't be generated in Unity Editor(with or without remote to device), but from observer.IsRunning it shows it's at least trying, so probably a bug in Editor?

By the way, I have a project in MRTK 2.5 which works fine in both Unity Editor, remote or deployed.

[wallacewangchao]

I am using MRTK 2.7.3 + OpenXR + Unity 2020.3 and 2021.2 I also have the same issue. I found the code starts working after I leave only "OpenXR Spatial Mesh Observer" in the settings, and I need to deploy to my device. (Maybe in the new MRTK version it needs to specify which observer is being access to? are we missing a document for that, or it's a bug?) Seems spatial mesh won't be generated in Unity Editor(with or without remote to device), but from observer.IsRunning it shows it's at least trying, so probably a bug in Editor?

By the way, I have a project in MRTK 2.5 which works fine in both Unity Editor, remote or deployed.

Thanks a lot! It works when I added "OpenXR Spatial Mesh Observer"

[keveleigh]

Seems spatial mesh won't be generated in Unity Editor(with or without remote to device), but from observer.IsRunning it shows it's at least trying, so probably a bug in Editor?

This should work in-editor with remoting in all recent Mixed Reality OpenXR Plugin versions. If you're on the latest (1.4.0) and it's still not working, please let me know!

[mcoy001]

Hi, I have tried 1.4.0, spital scan still doesn't work with remote. Also start play will launch Steam VR even if I have exit Steam. not sure if it's related, I got this from debug: Tracking origin unable to be set. Failed to calculate boundary bounds.

In the mean time my old project using Unity2019.4.31f1 with MRTK2.5.3 works fine with remote.

[DuncanHA]

I'm getting the exact same issue on Unity 2021.3.9f1 and MRTK 2.9.2. The solution supplied does seem to work but the editor errors are annoying and I can't get a 'pulse mesh' working with OpenXR. Any updates would be appreciated.

[julierthanjulie]

I'm still getting the same issue "Tracking Origin unable to be set" and "failed to calculate boundary bounds" after added the open XR spatial mesh observer. I'm using Hololens 2.

Even the sample code "out of the box" can't initialise the QR code tracking. Any updates?

[IssueSyncBot]

We appreciate your feedback and thank you for reporting this issue.

Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.

Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.

Please note that MRTK3 was released in August 2023. It features an all-new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visit https://www.mixedrealitytoolkit.org.

Thank you for your continued support of the Mixed Reality Toolkit!