Rogue mesh for OpenXR spatial mesh observer

[ShayneRC]

Describe the bug

We have some rogue mesh appearing. It looks like chunks of mesh that are separate from the rest of the main spatial observer mesh. It appears separate because when we enable/disable or switch to occlusion/wireframe, it isn't affected and remains there. Sometime the mesh is the no-shader pink and sometimes it is just invisible. The main way we can recognise it is by the hand ray colliding with it. The object the hand ray collides with is called SpatialMesh - xxxxxxxxx.

They have appeared as small as 10cm to 3-4m large. I believe the only way to remove them is by resetting the holograms in the OS. We have tried CoreServices.SpatialAwarenessSystem.ClearObservations() and CoreServices.SpatialAwarenessSystem.Reset(), neither of those have worked.

The first image attached shows the hand ray colliding with the mesh when we have the spatial awareness system disabled (CoreServices.SpatialAwarenessSystem.Disable()). The second image is when it's visible with the no-shader pink, it is mixed in with a hologram.

It seems these appear more often when doing an app version upgrade. E.g BIMHolovew-v2.2.2 -> BIMHoloview-v2.2.3.

To reproduce

Steps to reproduce the behavior:

1. Have an OpenXR Unity build with these MRTK Spatial Awareness settings and OpenXR Plugin settings:

2. Build and deploy to HoloLens 2.
3. We haven't narrowed down the exact steps to reproduce this issue. It appears to be random at the moment.

Expected behavior

They should not appear or they should change based on CoreServices.SpatialAwarenessSystem.

Screenshots

Your setup (please complete the following information)

• Unity Version 2020.3.13f1
• MRTK Version v2.7.2
• OpenXR Plugin Version v1.2.8
• Mixed Reality OpenXR Plugin v1.1.2

Target platform (please complete the following information)

• HoloLens 2

Additional context

We never experienced this with the Windows Mixed Reality Plugin or earlier versions of Unity/MRTK.

[peterneil-bimholoview]

I work with Shayne and have found another case where instead of pink or invisible mesh showing, actual mesh is showing as the rogue artifact. The first image is with the Spatial Observer enabled. The second is with it disabled and there are a few rogue mesh artifacts left.

[polar-kev]

@peterneil-bimholoview, is this something you're able to reproduce? @ShayneRC mentioned that the issue with the pink mesh is random.

[peterneil-bimholoview]

Unfortunately I am not what conditions cause this to happen. Essentially we enable the spatial observer, then (after user action) we disable the spatial observer and sometime after that the rouge mesh sometime appear. If it is straight after the disable then we will generally see mesh (bottom case), but if it occurs later in the session then the mesh will be 'no shader pink' or colorless.

[david-c-kline]

This is something I recall addressing for the legacy spatial mesh observer. i am trying to repro now

[david-c-kline]

while i get the project loaded, i thought i would jot down a few notes..

this feels as if there is a mesh bake in flight and the callback is not honoring the flag that indicates whether or not the observer should be running. the 'no shader pink' condition is a bit puzzling, but is something i will definitely look for cases where there may be confusion regarding which material to apply

[david-c-kline]

looking at the code, if my hunch is correct, this should impact all spatial mesh observers shipped in MRTK

[peterneil-bimholoview]

We only have seen this with the OpenXR plugin, and we did a lot of testing with the WMR Plugin before switching over.

[peterneil-bimholoview]

Just seen this with WMR Plugin.

[polar-kev]

@peterneil-bimholoview, was this still a random event or is it happening consistently now?

[peterneil-bimholoview]

@polar-kev still intermittent. This was the first time I have seen in with WMR, but with OpenXR it occurs in about 1;4 application sessions. However, it may relate to the Observer Shape being User Aligned Cube rather than the plugin type. Our OpenXR version is running User Aligned Cube, and we just switched our WMR version from Axis Aligned Cube to User Aligned Cube and saw the issue. But I need to do some more testing on this to confirm.

[MaxWang-MS]

@peterneil-bimholoview Just to clarify is this the same as #10343 or these two issues are different?

[peterneil-bimholoview]

@MaxWang-MS No different issue. The Rogue mesh issue is where you mesh turns up (generally Pink) when the observer is off. Issue #10343 is when the observation extents becomes offset due to world locking, which I can avoid by turning off the WLT Auto-load feature.

[peterneil-bimholoview]

@polar-kev Just confirming tests from today. The issue appears in both User Aligned Cube & Axis Aligned Cube.

[Gescheld]

@peterneil-bimholoview, was this still a random event or is it happening consistently now?

I have made the observation that it always occurs to me when I change my Level of Detail to custom at runtime, plus the density (Triangle per Cubic Meter) and then restart the observer. Changing only to custom does nothing. Only by additionally changing the density. (With WMR). The oberver is reseted, cleared and restarted after the changes at runtime by the following: CoreServices.SpatialAwarenessSystem.Reset(); CoreServices.SpatialAwarenessSystem.ClearObservations(); CoreServices.SpatialAwarenessSystem.ResumeObservers();

Maybe this will help to find the bug...

[peterneil-bimholoview]

@Gescheld I am good to hear someone else is seeing this issue - not just us!!!

We don't change the density but we do set the DisplayOption when we Enable the Observer. I wonder if this is issue only appears if you Enable/Disable plus set another property on the observer?

[Gescheld]

@peterneil-bimholoview, I had only tried from default visible and LoD coarse to not visible and swappable LoD with changeable Triangle per Cubic Meter. While continuing to work, I changed my package.appxmanifest due to another task. And I noticed that in my script, where I change the observer, I refer to an unassigned text element (but without error message and the program ran anyway). Since then, the pink mesh has not yet reappeared. But as it is with programming, when something doesn't work, you don't know why. And when it works, you don't know why either :D This is part of my package.appxmanifest:

<Package 
    xmlns:mp="http://schemas.microsoft.com/appx/2014/phone/manifest"
    xmlns:uap="http://schemas.microsoft.com/appx/manifest/uap/windows10"
    xmlns:uap2="http://schemas.microsoft.com/appx/manifest/uap/windows10/2"
    xmlns:uap3="http://schemas.microsoft.com/appx/manifest/uap/windows10/3"
    xmlns:uap4="http://schemas.microsoft.com/appx/manifest/uap/windows10/4"
    xmlns:iot="http://schemas.microsoft.com/appx/manifest/iot/windows10"
    xmlns:mobile="http://schemas.microsoft.com/appx/manifest/mobile/windows10"
    xmlns:rescap="http://schemas.microsoft.com/appx/manifest/foundation/windows10/restrictedcapabilities" (NEW)
    IgnorableNamespaces="uap uap2 uap3 uap4 mp mobile iot rescap" (rescap is NEW)
    xmlns="http://schemas.microsoft.com/appx/manifest/foundation/windows10">

and

<Capabilities>
    <Capability Name="internetClient" />
    <Capability Name="internetClientServer" />
    <Capability Name="privateNetworkClientServer" />
    <uap2:Capability Name="spatialPerception" />
    <rescap:Capability Name="perceptionSensorsExperimental" /> (NEW)
    <DeviceCapability Name="backgroundSpatialPerception"/>  (NEW)
    <DeviceCapability Name="webcam" />
    <DeviceCapability Name="microphone" />
    <DeviceCapability Name="gazeinput" />
</Capabilities>

[IssueSyncBot]

We appreciate your feedback and thank you for reporting this issue.

Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.

Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.

Please note that MRTK3 was released in August 2023. It features an all new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visithttps://www.mixedrealitytoolkit.org.

Thank you for your continued support of the Mixed Reality Toolkit!