Spatial Anchor saving sample doesn't work with OpenXR plugin

[robbbbbb]

Note: May be related to #8

Description

When switching to the OpenXR Plugin the Spatial Anchor scene sample is no longer able to save anchors. This is the same regardless of leaving the sample saving locally or switching to OVRSpace.StorageLocation.Cloud.

Using the Oculus XR Plugin both Local and Cloud saving works as expected.

Setup

Unity: 2021.3.36f1 Packages: Meta XR Core SDK v 62.0.0; OpenXR Plugin 1.10.0; XR Plugin Management 4.4.0 Device: Quest 3, OS version 63.0.0.399.366

Repro

With no other modifications to the code or project configuration:

• Install the OpenXR Plugin package
• Under Project Settings > XR Plug-in Management switch Plug-in Provider to OpenXR
• Build using Oculus > Samples > Build Starter Scene
• Deploy to device
• Select Spatial Anchor from the start scene menu
• Create an anchor
• Select the anchor
• Attempt to Save anchor

Expected Results

Anchor saves, green icon is shown and on re-running the app and selecting Load Anchors the anchor is loaded in the same location with the same GUID.

Actual Results

Nothing happens and green icon doesn't show. On re-running the app and selecting Load Anchors the previously created anchor is not visible.

Checking the logs the following error is printed when attempting to save the anchor: [SaveSpaceList] m_XR_FB_spatial_entity_storage_batch extension is not available (arvr/projects/integrations/OVRPlugin/Src/Util/CompositorOpenXR.cpp:12734)

[tdmowrer]

Checking the logs the following error is printed when attempting to save the anchor: [SaveSpaceList] m_XR_FB_spatial_entity_storage_batch extension is not available (arvr/projects/integrations/OVRPlugin/Src/Util/CompositorOpenXR.cpp:12734)

This could be a couple of things:

1. This extension is only available in newer versions of the OpenXR plugin. If I remember correctly, it requires 1.8 or later. What version of com.unity.xr.openxr do you have installed?
2. This extension requires an AndroidManifest entry. There is a project-level setting in the OculusProjectConfig (UI also available on the OVRManager). Make sure "Anchor Support" is set to "Enabled". If anchor saving works with the Oculus XR Plugin, then this is probably already the case, but mentioning it just in case.

[robbbbbb]

Hi, thanks for the response, but I don't think either of those things are the issue. We're on 1.10 of the OpenXR plugin (I did mention this in the original issue description) and the setting is enabled (like you say, it wouldn't work with the Oculus plugin otherwise.

[robbbbbb]

@tdmowrer your points earlier today got us looking in to the packages involved again. It seems that the OpenXR plugin itself doesn't really have very much to do with the availability or not of this feature (beyond the fact that it provides the system for defining features themselves).

The actual extensions are enabled by the Meta XR Feature group, which in turn is implemented in the Meta XR Core SDK package (aka com.meta.xr.sdk.core, we're using v63.0.0 but that's largely irrelevant as we'll see)...

If you look in this package you'll find the file Scripts/OpenXRFeatures/MetaXRFeature.cs which contains an OpenXRFeature definition with an OpenxrExtensionStrings property set to a giant list of OpenXR extensions. This list presumably just gets poked through to the actual OpenXR API implementation on device.

That list does contain a bunch of extensions starting with "XR_FB_spatial_entity..." but vitally does not contain the specific one the sample is failing on: XR_FB_spatial_entity_storage_batch.

To our surprise, simply copying the package from the cache in to the project and adding this extension string suddenly makes everything work.

Specific steps:

• Follow project setup as per reproduction steps in this issue above...
• Copy the entire com.meta.xr.sdk.core@63.0.0 folder from ./Library/PackageCache to ./Packages (the Unity package manager will update the reference automatically)
• Edit the file ./Packages/com.meta.xr.sdk.core@63.0.0/Scripts/OpenXRFeatures/MetaXRFeature.cs and add this line anywhere in the definition of the OpenXrExtensionList string: "XR_FB_spatial_entity_storage_batch " +
• Build, deploy and follow same repro steps

We're pretty sure this is also what's causing this issue in the Unity Spatial Anchors sample repo https://github.com/oculus-samples/Unity-SharedSpatialAnchors/issues/19

And an equivalent missing feature declaration also appears to be causing multiple other platform features to not work either as expected or at all when using OpenXR. As you can see some of this definitely an issue with the OpenXR plugin package, but some is definitely a problem with the Meta XR Core SDK package, so it would be good if someone in Meta could take a look and get the most up to date feature set working with the OpenXR plugin since it seems to be a question of a few missing feature declarations.

[robbbbbb]

Since we're on the topic of what's on Unity's plate vs Meta's, the Unity OpenXR plugin documentation has a special section talking about Meta Quest Support https://docs.unity3d.com/Packages/com.unity.xr.openxr@1.11/manual/features/metaquest.html. That section explicitly links to a Meta blog post that is coming up on 3 years old which implies two things:

1. That the OpenXR Plugin will be "fully supported in early 2022"
2. That the Oculus XR Plugin will go out of support in early 2023

It feels like neither of these things has happened. If anything, the OpenXR plugin remains not very well supported, and there's an implicit assumption that the Oculus XR plugin is the only one really being used by anyone.

It'd be helpful for Unity, Meta, or even both to put some kind of updated version of that 3 year old blog post to help developers understand what they're supposed to be doing.

[tdmowrer]

That list does contain a bunch of extensions starting with "XR_FB_spatial_entity..." but vitally does not contain the specific one the sample is failing on: XR_FB_spatial_entity_storage_batch.

After I wrote my response yesterday, I went down the same path and came to the same conclusion. It is indeed missing the storage_batch extension. This will be fixed in a future release, but in the meantime, you don't need to embed & modify the Meta XR Core SDK. Instead, you can add OpenXR extensions yourself (docs) with something like this:

using UnityEngine;
using UnityEngine.XR.OpenXR.Features;

#if UNITY_EDITOR
using UnityEditor;
using UnityEditor.XR.OpenXR.Features;
#endif

#if UNITY_EDITOR
[OpenXRFeature(
    UiName = "Anchor Storage Batch",
    BuildTargetGroups = new[] { BuildTargetGroup.Standalone, BuildTargetGroup.Android },
    Company = "My Company",
    Desc = "OpenXR extension additions for anchors",
    DocumentationLink = "",
    OpenxrExtensionStrings = "XR_FB_spatial_entity_storage_batch ",
    Version = "0.0.1",
    FeatureId = featureId)]
#endif
public class CustomAnchorFeature : OpenXRFeature
{
    public const string featureId = "com.mycompany.openxr.anchors";
}

#if UNITY_EDITOR
[OpenXRFeatureSet(
    FeatureIds = new[]
    {
        CustomAnchorFeature.featureId
    },
    UiName = "My Custom Additions",
    Description = "OpenXR extension additions",
    FeatureSetId = featureSetId,
    SupportedBuildTargets = new[] { BuildTargetGroup.Android, BuildTargetGroup.Standalone },
    RequiredFeatureIds = new[]
    {
        CustomAnchorFeature.featureId,
    },
    DefaultFeatureIds = new[]
    {
        CustomAnchorFeature.featureId,
    }
)]
sealed class MyCustomOpenXRFeatureSet
{
    public const string featureSetId = "com.mycompany.openxr.featureset";
}
#endif

Then, just enable your new "feature group":

You'll know it works if you see the storage_batch extension listed under Runtime extensions enabled in logcat:

Runtime extensions enabled: (42)
...
  XR_FB_spatial_entity: Version=1
  XR_FB_spatial_entity_container: Version=1
  XR_FB_spatial_entity_query: Version=1
  XR_FB_spatial_entity_storage: Version=1
  XR_FB_spatial_entity_storage_batch: Version=1
...

[robbbbbb]

Thanks @tdmowrer, that's for sure a more elegant fix, and thanks for taking the time to detail it all out :)

My comment though was really just trying to illustrate that it seems to simply be an oversight in the Core SDK, rather than any more fundamental problem with the Unity core engine code, the OpenXR plugin, etc... (i.e. it's something that's purely a Meta thing, not a dependency on Unity), so it's weird and kind of frustrating that it doesn't just work given that there are multiple reports of it like this one on github https://github.com/oculus-samples/Unity-SharedSpatialAnchors/issues/19 or this one on the developer forums https://communityforums.atmeta.com/t5/Unity-VR-Development/Share-Spatial-Anchors-OpenXR-Unity/td-p/1137482, neither of which has had any feedback 😞

There are also other features which aren't supported (sorry to hijack this but there's no more appropriate venue to report bugs like this, so just in case you have a line through...):

• The OVRManager.IsPassthroughRecommended feature doesn't work with the OpenXR plugin, it doesn't generate an error, it just returns false regardless of the OS state. This can be fixed by just registering the XR_META_passthrough_preferences extension string in a similar manner as above.
• OVROverlays with Depth Buffer Testing enabled. Again there's no error, the compositing just doesn't work and the overlay is always on top, so no handy clue as to what the extension might be unlike with the anchor storage batch extension. We attempted just registering the XR_FB_composition_layer_depth_test extension as a guess, but no dice. I think in this case we may need more than the extensions correctly activated, and it's actually to do with the 'Depth Submision (Vulkan)' flag which is available on the Oculus XR plugin options but has no obvious (Vulkan-specific) equivalent in the OpenXR plugin options (the closest is just the generic 'Depth Submission Mode' which has no effect on this). I reported this in this issue https://github.com/oculus-samples/Unity-StarterSamples/issues/13 which you may have already spotted.

Again, I appreciate this is an issue on a samples repo, not a generic dumping ground for Quest SDK issues, but we don't seem to have a lot of options and you've made the fatal error of being helpful on the internet! 💀

[tdmowrer]

it seems to simply be an oversight in the Core SDK, rather than any more fundamental problem with the Unity core engine code

Yep, I agree, and it will be fixed in the Core SDK (I work on the Core SDK). We're going to make sure we aren't missing any more, like XR_META_passthrough_preferences.

The OVROverlay issue is probably different; I'll respond on #13.

Thanks for the the very detailed reports!

[robbbbbb]

@tdmowrer thank you, that’s great news!

We have a couple of other things that aren’t working as expected related to the OpenXR plugin vs Oculus XR, happy to dump them in this thread, create other issues on this repo, or raise tickets/issues somewhere else if that’s more appropriate? Just let us know what’s most helpful.

[tdmowrer]

We have a couple of other things that aren’t working as expected related to the OpenXR plugin vs Oculus XR, happy to dump them in this thread, create other issues on this repo, or raise tickets/issues somewhere else if that’s more appropriate?

Please create a separate issue (this should help focus this thread for future readers). Thanks for the great bug reports; keep them coming!

[ASquareDevs]

Not to hijack the thread again, but curious how using Spatial Anchors comes into play with OpenXR and Shared Spatial Anchors. To robbbbbb's point, I've been jumping through a dozen different sites, announcements, APIs, etc and not seeing conclusive answers.

Are Shared Spatial Anchors supported in OpenXR? This troubleshooting guide briefly states if OpenXR is being used, here is how to resolve an issue, but how do we link the App ID to allow for sharing point cloud data? Do we still have to do this step? Is it even supported? I can't find any answers from the OpenXR side.

From this issue tracker it looks like other devs are using the Meta XR Core SDK to link the App ID for Shared Spatial Anchors.

Would love some answers if possible. Thanks!

[tdmowrer]

@ASquareDevs Although a related feature, I don't think what you've described has the same root cause. Please create a separate issue -- this helps focus the thread for future readers, and a thread marked "closed" is less likely to receive attention.