Azure / azure-spatial-anchors-samples

Azure Spatial Anchors sample code
Other
293 stars 138 forks source link

Azure Anchors not working on UWP x64 OpenXR Remoting Build #381

Closed ikto-art closed 1 year ago

ikto-art commented 1 year ago

Description

I can't use the SpatialAnchorManager script in Unity when I try to use it in a build, after establishing a connection to a HoloLens 2 with OpenXR remoting.

SpatialPerception capability is not enabled
System.InvalidOperationException: SpatialAnchorManager is improperly configured.

Steps to reproduce the issue

Basic project which uses the Mixed Reality OpenXR 2.8.1 remoting runtime and MRTK 2.8.2 on Unity 2020.3.41 with UWP build. When running built player and calling SpatialAnchorManager.CreateSessionAsync() after the remoting session is established it says SpatialPerception capability not enabled (of course the package manifest has the SpatialPerception authorization).

Development information (please complete the following information)

AR Device information (please complete the following information):

Additional context

In the UWP build, a script connects to the HoloLens 2 with AppRemoting.Connect() then when connected waits 3 seconds for OpenXR sessions status to go to 'Focused' and then tries to create and start a session with SpatialAnchorManager.CreateSessionAsync().

In the log I have the following report before the call to Windows.Perception.Spatial.SpatialAnchorExporter.RequestAccessAsync():

==== Start Unity OpenXR Diagnostic Report ====

[XR] [29208] [04:57:58.427][Info   ] ==== Report Reason: System Startup Completed ====

[XR] [29208] [04:57:58.427][Info   ] ==== OpenXR Init Info ====
[XR] [29208] [04:57:58.427][Info   ] 
[XR] [29208] [04:57:58.427][Info   ] Stage 1 Missing Entry Points: (1)
[XR] [29208] [04:57:58.427][Info   ]   xrInitializeLoaderKHR
[XR] [29208] [04:57:58.427][Info   ] 
[XR] [29208] [04:57:58.427][Info   ] Stage 2 Missing Entry Points: (1)
[XR] [29208] [04:57:58.427][Info   ]   xrGetD3D12GraphicsRequirementsKHR
[XR] [29208] [04:57:58.427][Info   ] 
[XR] [29208] [04:57:58.427][Info   ] ==== OpenXR Provider Info ====
[XR] [29208] [04:57:58.427][Info   ] 
[XR] [29208] [04:57:58.427][Info   ] Spec Version: 1.0.23
[XR] [29208] [04:57:58.427][Info   ] Provider Version: 1.5.3
[XR] [29208] [04:57:58.427][Info   ] App: HL2020Streaming #3159629941
[XR] [29208] [04:57:58.427][Info   ] Engine: 2020.3.41f1 #20091905
[XR] [29208] [04:57:58.427][Info   ] 
[XR] [29208] [04:57:58.427][Info   ] ==== OpenXR Runtime Info ====
[XR] [29208] [04:57:58.427][Info   ] 
[XR] [29208] [04:57:58.427][Info   ] Features requested to be enabled: (7)
[XR] [29208] [04:57:58.427][Info   ]   Eye Gaze Interaction Profile: Version=0.0.1, Company="Unity", Extensions="XR_EXT_eye_gaze_interaction"
[XR] [29208] [04:57:58.427][Info   ]   Hand Tracking: Version=1.6.0, Company="Microsoft", Extensions="XR_EXT_hand_tracking XR_EXT_hand_joints_motion_range XR_MSFT_hand_tracking_mesh"
[XR] [29208] [04:57:58.427][Info   ]   Microsoft Hand Interaction Profile: Version=0.0.1, Company="Unity", Extensions="XR_MSFT_hand_interaction"
[XR] [29208] [04:57:58.427][Info   ]   Microsoft Motion Controller Profile: Version=0.0.1, Company="Unity"
[XR] [29208] [04:57:58.427][Info   ]   Mixed Reality Features: Version=1.6.0, Company="Microsoft", Extensions="XR_MSFT_holographic_window_attachment XR_KHR_win32_convert_performance_counter_time XR_MSFT_unbounded_reference_space XR_MSFT_spatial_anchor XR_MSFT_secondary_view_configuration XR_MSFT_first_person_observer XR_MSFT_spatial_graph_bridge XR_MSFT_perception_anchor_interop XR_MSFT_spatial_anchor_persistence XR_MSFT_scene_understanding XR_MSFT_scene_understanding_serialization XR_MSFT_spatial_anchor_export_preview XR_MSFT_composition_layer_reprojection"
[XR] [29208] [04:57:58.427][Info   ]   Motion Controller Model: Version=1.6.0, Company="Microsoft", Extensions="XR_MSFT_controller_model XR_FB_render_model"
[XR] [29208] [04:57:58.427][Info   ]   Holographic Remoting remote app: Version=1.6.0, Company="Microsoft", Extensions="XR_MSFT_holographic_remoting XR_MSFT_holographic_remoting_speech"
[XR] [29208] [04:57:58.427][Info   ] 
[XR] [29208] [04:57:58.427][Info   ] Requested feature extensions not supported by runtime: (9)
[XR] [29208] [04:57:58.427][Info   ]   XR_EXT_hand_joints_motion_range: Feature="Hand Tracking": Version=1.6.0, Company="Microsoft"
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_hand_tracking_mesh: Feature="Hand Tracking": Version=1.6.0, Company="Microsoft"
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_holographic_window_attachment: Feature="Mixed Reality Features": Version=1.6.0, Company="Microsoft"
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_secondary_view_configuration: Feature="Mixed Reality Features": Version=1.6.0, Company="Microsoft"
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_first_person_observer: Feature="Mixed Reality Features": Version=1.6.0, Company="Microsoft"
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_perception_anchor_interop: Feature="Mixed Reality Features": Version=1.6.0, Company="Microsoft"
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_spatial_anchor_export_preview: Feature="Mixed Reality Features": Version=1.6.0, Company="Microsoft"
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_composition_layer_reprojection: Feature="Mixed Reality Features": Version=1.6.0, Company="Microsoft"
[XR] [29208] [04:57:58.427][Info   ]   XR_FB_render_model: Feature="Motion Controller Model": Version=1.6.0, Company="Microsoft"
[XR] [29208] [04:57:58.427][Info   ] 
[XR] [29208] [04:57:58.427][Info   ] Available Runtime Extensions: (21)
[XR] [29208] [04:57:58.427][Info   ]   XR_KHR_win32_convert_performance_counter_time: Version=1
[XR] [29208] [04:57:58.427][Info   ]   XR_KHR_D3D11_enable: Version=5
[XR] [29208] [04:57:58.427][Info   ]   XR_KHR_D3D12_enable: Version=7
[XR] [29208] [04:57:58.427][Info   ]   XR_KHR_composition_layer_depth: Version=5
[XR] [29208] [04:57:58.427][Info   ]   XR_KHR_composition_layer_color_scale_bias: Version=5
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_unbounded_reference_space: Version=1
[XR] [29208] [04:57:58.427][Info   ]   XR_EXT_win32_appcontainer_compatible: Version=1
[XR] [29208] [04:57:58.427][Info   ]   XR_EXT_eye_gaze_interaction: Version=1
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_hand_interaction: Version=1
[XR] [29208] [04:57:58.427][Info   ]   XR_EXT_hand_tracking: Version=4
[XR] [29208] [04:57:58.427][Info   ]   XR_KHR_visibility_mask: Version=2
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_spatial_anchor: Version=2
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_spatial_anchor_persistence: Version=2
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_spatial_graph_bridge: Version=1
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_scene_understanding: Version=1
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_scene_understanding_serialization: Version=1
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_holographic_remoting: Version=3
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_holographic_remoting_frame_mirroring: Version=1
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_controller_model: Version=2
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_holographic_remoting_speech: Version=3
[XR] [29208] [04:57:58.427][Info   ]   XR_EXT_debug_utils: Version=4
[XR] [29208] [04:57:58.427][Info   ] 
[XR] [29208] [04:57:58.427][Info   ] Available Layers: (0)
[XR] [29208] [04:57:58.427][Info   ] 
[XR] [29208] [04:57:58.427][Info   ] Runtime Name: Microsoft Holographic AppRemoting Runtime
[XR] [29208] [04:57:58.427][Info   ] Runtime Version: 2.8.1
[XR] [29208] [04:57:58.427][Info   ] 
[XR] [29208] [04:57:58.427][Info   ] Form factor: XR_FORM_FACTOR_HEAD_MOUNTED_DISPLAY
[XR] [29208] [04:57:58.427][Info   ] 
[XR] [29208] [04:57:58.427][Info   ] Runtime extensions enabled: (16)
[XR] [29208] [04:57:58.427][Info   ]   XR_EXT_eye_gaze_interaction: Version=1
[XR] [29208] [04:57:58.427][Info   ]   XR_EXT_hand_tracking: Version=4
[XR] [29208] [04:57:58.427][Info   ]   XR_KHR_D3D11_enable: Version=5
[XR] [29208] [04:57:58.427][Info   ]   XR_KHR_composition_layer_depth: Version=5
[XR] [29208] [04:57:58.427][Info   ]   XR_KHR_visibility_mask: Version=2
[XR] [29208] [04:57:58.427][Info   ]   XR_KHR_win32_convert_performance_counter_time: Version=1
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_controller_model: Version=2
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_hand_interaction: Version=1
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_holographic_remoting: Version=3
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_holographic_remoting_speech: Version=3
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_scene_understanding: Version=1
[XR] [29208] [04:57:58.427][Info   ]   XR_MSFT_scene_understanding_serialization: Version=1
[XR] [29208] [04:57:58.428][Info   ]   XR_MSFT_spatial_anchor: Version=2
[XR] [29208] [04:57:58.428][Info   ]   XR_MSFT_spatial_anchor_persistence: Version=2
[XR] [29208] [04:57:58.428][Info   ]   XR_MSFT_spatial_graph_bridge: Version=1
[XR] [29208] [04:57:58.428][Info   ]   XR_MSFT_unbounded_reference_space: Version=1
[XR] [29208] [04:57:58.428][Info   ] 
[XR] [29208] [04:57:58.428][Info   ] Available Reference Spaces: (4)
[XR] [29208] [04:57:58.428][Info   ]   XR_REFERENCE_SPACE_TYPE_VIEW
[XR] [29208] [04:57:58.428][Info   ]   XR_REFERENCE_SPACE_TYPE_LOCAL
[XR] [29208] [04:57:58.428][Info   ]   XR_REFERENCE_SPACE_TYPE_UNBOUNDED_MSFT
[XR] [29208] [04:57:58.428][Info   ]   Unknown XrReferenceSpaceType
[XR] [29208] [04:57:58.428][Info   ] 
[XR] [29208] [04:57:58.428][Info   ] Instance Id  : 0x1
[XR] [29208] [04:57:58.428][Info   ] System Id    : 0x1
[XR] [29208] [04:57:58.428][Info   ] Session Id   : 0x2
[XR] [29208] [04:57:58.428][Info   ] 
[XR] [29208] [04:57:58.428][Info   ] Available View Configuration Types: (1)
[XR] [29208] [04:57:58.428][Info   ]   XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO (Selected): FovMutable=True
[XR] [29208] [04:57:58.428][Info   ]     [0]: Recommended: Width=1440, Height=936, SampleCount=1
[XR] [29208] [04:57:58.428][Info   ]     [0]:     Maximum: Width=16384, Height=16384, SampleCount=8
[XR] [29208] [04:57:58.428][Info   ]     [1]: Recommended: Width=1440, Height=936, SampleCount=1
[XR] [29208] [04:57:58.428][Info   ]     [1]:     Maximum: Width=16384, Height=16384, SampleCount=8
[XR] [29208] [04:57:58.428][Info   ] 
[XR] [29208] [04:57:58.428][Info   ] Skip Present To Main Screen: No
[XR] [29208] [04:57:58.428][Info   ] 
[XR] [29208] [04:57:58.428][Info   ] Available Environment Blend Modes: (1)
[XR] [29208] [04:57:58.428][Info   ]   XR_ENVIRONMENT_BLEND_MODE_ADDITIVE (Selected)
[XR] [29208] [04:57:58.428][Info   ] 
[XR] [29208] [04:57:58.428][Info   ] System Properties: Name="Remoting Xr System", VendorId=1118
[XR] [29208] [04:57:58.428][Info   ] System Graphics Properties: MaxWidth=16384, MaxHeight=16384, MaxLayers=16
[XR] [29208] [04:57:58.428][Info   ] System Tracking Properties: OrientationTracking=True, PositionTracking=True
[XR] [29208] [04:57:58.428][Info   ] 
[XR] [29208] [04:57:58.428][Info   ] ==== OpenXR Input ====
[XR] [29208] [04:57:58.428][Info   ] 
[XR] [29208] [04:57:58.428][Info   ] Action Sets (3): 
[XR] [29208] [04:57:58.428][Info   ]   eyegaze: ActionCount=1
[XR] [29208] [04:57:58.428][Info   ]   microsofthandinteraction: ActionCount=6
[XR] [29208] [04:57:58.428][Info   ]   microsoftmotioncontroller: ActionCount=13
[XR] [29208] [04:57:58.428][Info   ] 
[XR] [29208] [04:57:58.428][Info   ] 
[XR] [29208] [04:57:58.428][Info   ] ==== OpenXR Swapchain Details ====
[XR] [29208] [04:57:58.428][Info   ] 
[XR] [29208] [04:57:58.428][Info   ] Render Mode: Single Pass Instanced
[XR] [29208] [04:57:58.428][Info   ] Depth Submission Mode: Depth16Bit
[XR] [29208] [04:57:58.428][Info   ] 
[XR] [29208] [04:57:58.428][Info   ] Swapchain Formats:  [c:91] 29 87 28 40 20 45 [d:55]
[XR] [29208] [04:57:58.428][Info   ] Swapchain Views: (2)
[XR] [29208] [04:57:58.428][Info   ]   [0]: Width=1440, Height=936, SampleCount=1
[XR] [29208] [04:57:58.428][Info   ] 
[XR] [29208] [04:57:58.428][Info   ] ==== Last 20 non-XR_SUCCESS returns ====
[XR] [29208] [04:57:58.428][Info   ] [SUCCESS] xrGetReferenceSpaceBoundsRect: XR_SPACE_BOUNDS_UNAVAILABLE (2x)
[XR] [29208] [04:57:58.428][Info   ] 
[XR] [29208] [04:57:58.428][Info   ] ==== Last 20 Events ====
[XR] [29208] [04:57:58.428][Info   ] XrEventDataSessionStateChanged: XR_SESSION_STATE_UNKNOWN->XR_SESSION_STATE_IDLE
[XR] [29208] [04:57:58.428][Info   ] XrEventDataSessionStateChanged: XR_SESSION_STATE_IDLE->XR_SESSION_STATE_READY
[XR] [29208] [04:57:58.428][Info   ] XrEventDataSessionStateChanged: XR_SESSION_STATE_READY->XR_SESSION_STATE_SYNCHRONIZED
[XR] [29208] [04:57:58.428][Info   ] XrEventDataSessionStateChanged: XR_SESSION_STATE_SYNCHRONIZED->XR_SESSION_STATE_VISIBLE
[XR] [29208] [04:57:58.428][Info   ] XrEventDataSessionStateChanged: XR_SESSION_STATE_VISIBLE->XR_SESSION_STATE_FOCUSED
[XR] [29208] [04:57:58.428][Info   ] 

==== End Unity OpenXR Diagnostic Report ====

I see this extension that might be needed that's not supported: XR_MSFT_spatial_anchor_export_preview But the 2.8.1 remoting library changelog seems to imply that Azure Spatial Anchors are working with Remoting.

ikto-art commented 1 year ago

I also tried ignoring the validation checks in SpatialAnchorManager.cs and got this error from ASA: System.InvalidOperationException: Azure Spatial Anchors requires xrConvertWin32PerformanceCounterToTimeKHR, xrTryGetPerceptionAnchorFromSpatialAnchorMSFT, xrCreateSpatialAnchorFromPerceptionAnchorMSFT, xrCreateSpatialAnchorMSFT, and xrReadSpatialAnchorNeighborhoodDataMSFT extensions avaialble.

I forgot to write earlier that the first time I ran the UWP build it seemed to not be able to find an available XR runtime or connect to the HoloLens 2 so I added one manually that pointed to Microsoft.Holographic.AppRemoting.OpenXr.dll and then it worked. Now I just tried to install the Microsoft app 'OpenXR for Windows Mixed Reality' and set it as the available runtime but in the log there were errors of XR_ERROR_FORM_FACTOR_UNAVAILABLE then it apparently automatically switched to the remoting runtime that still lacks the extensions (but the Microsoft app runtime had the XR_MSFT_perception_anchor_interop extension).

dfields-msft commented 1 year ago

@ikto-art your observations are right on track - OpenXR Remoting 2.8.1 indeed has the necessary platform support for ASA to work, but the ASA SDK still depends on a runtime-specific extension that's only present in the non-Remoting runtime. Once that dependency is removed, the ASA SDK should work with OpenXR Remoting 2.8.1 and higher.

ikto-art commented 1 year ago

Thank you for that information. Do you know approximately when that might happen? Also, when you say the remoting runtime has the necessary platform support you mean users of that runtime have the access to export anchors or is it a private access for now? I wanted to try the XRAnchorTransferBatch class before replying to your comment but didn't have the time yet, I'll try it as soon as possible.

dfields-msft commented 1 year ago

The ASA SDK changes to enable this should be checked in shortly and published in the next 1-2 months (I'm not familiar with the exact schedule).

As for XRAnchorTransferBatch, this is not supported by Remoting and there are no plans to add this. More generally, anchor transfer batches have a number of significant drawbacks and should generally not be used, in favor of Azure Spatial Anchors. See the deprecation notes on the core API for anchor transfer batches: https://learn.microsoft.com/en-us/uwp/api/windows.perception.spatial.spatialanchortransfermanager?view=winrt-22621#remarks

ikto-art commented 1 year ago

Thank you, yes I knew it was deprecated but still the only workaround for now. I'll wait for the update.

Varnauld commented 1 year ago

Adding @darax who is working on the next SDK update (currently scheduled end of this month). Thank you

darax commented 1 year ago

this should work in 2.14.1.