search

Azure / azure-spatial-anchors-samples

Azure Spatial Anchors sample code
Other
293 stars 139 forks source link

Anchors not consistently found with OpenXR Unity plugin #312

Closed robbbbbb closed 2 years ago

robbbbbb commented 3 years ago

Description

When running the 2.10.2 SDK with the OpenXR plugin, the SpatialAnchorManager.AnchorLocated callback is often just never received for a valid anchor in situations where it should be. Shutting down the session and restarting it (without killing the app, changing environmental conditions, or moving the device in any significant way) will often result in the same anchor being found immediately. Subsequent repetitions of the same cycle of stop/starting the ASA session show non-deterministic results, with the anchor sometimes being found immediately and other times never being found.

When the anchor is not located, no errors are reported by the LogDebugor Error event, the AnchorLocated event simple never fires.

Note: I reported this as an issue on the OpenXR-Unity-MixedReality-Samples repo, but since I can repro it in our app (see environment setup below) I believe there's a pretty good chance it not an issue with those samples, but with either the ASA SDK, or with the OpenXR plugin or runtime.

Steps to reproduce the issue

Steps to reproduce the behavior:

  1. Build and run this OpenXR ASA Sample
  2. Tap 'Start Session'
  3. Place 2 anchors by air tapping
  4. Save both anchors by air tapping in their boxes
  5. Confirm both have been saved
  6. Tap 'Stop Session'
  7. Tap 'Start Session'
  8. Observe that both anchors were located and appear as green boxes
  9. Tap 'Stop Session'
  10. Tap 'Start Session'
  11. Observe that (usually) at least one, or both of the anchors are never located
  12. Repeat steps 9-11 several times (varying the wait between stopping and restarting the session if you want to), and observe that the ability of the SDK to locate the anchors is seemingly random (and usually fails).

Expected behavior

Under reasonable environmental conditions an ASA anchor in a given location should always be found, or at least some error logging should happen against the SpatialAnchorManager's Error callback.

Development information (please complete the following information)

AR Device information (please complete the following information):

msftradford commented 3 years ago

Hi @robbbbbb, thanks for reaching out! FYI 2.10.0-beta0014 was a prerelease version of the SDK. I recommend upgrading to the latest official release, 2.10.2, if you haven't already. Please let us know if you have similar issues after the upgrade. Thanks!

robbbbbb commented 3 years ago

Hi @msftradford, thanks for the reply.

I used 2.10.0-beta0014 because that was the version that the sample project in that repo was set to use (so I assumed it would work). I have also upgraded that sample project to 2.10.2 and tried our project with both 2.10.0-beta0014 and with 2.10.2 and the results are identical in all 4 cases. So it seems there's nothing improved in the official release.

Can you confirm whether you can repro the issue with the sample project?

robbbbbb commented 3 years ago

@msftradford any further thoughts on this?

There was a suggestion to try updating the Mixed Reality OpenXR package to the latest 1.1.1 version. I have done so (in our project, and in the OpenXR ASA Ssample project I mentioned). The consistency of anchor location was not improved. If anything it got worse.

Would be great to get any kind of feedback on this issue.

msftradford commented 3 years ago

Hey @robbbbbb thanks very much for the updates. I have been able to repro something similar to what you've described but I want to make sure that we're seeing the same thing.

Here are the steps that I took to set-up the project:

  1. Clean checkout of ASA+MROpenXR sample repo
  2. Open ASA sample project in Unity
  3. Remove ASA SDK 2.10.0-preview.1 Core and Windows packages
  4. Open project in MRFT and install ASA SDK 2.10.2 Core and Windows packages

Everything else should be the same from the clean checkout (so MROpenXR 1.0.3, MRTK 2.7.2, OpenXR 1.2.8, etc.). When I deploy the project with that combination, I have seen an issue where an anchor or anchors are not returned at their correct poses. The GameObjects associated with these anchors appear at the origin of the Unity world coordinate system (where your device was when the application started) and an additional log message prints out that it "didn't find the anchor". Is this what you are seeing?

I want to make sure because it sounds like you were implying that you are not receiving the SpatialAnchorManager.AnchorLocated event for some anchors. In my scenario, I do receive the event for all anchors, but they are returned at incorrect poses (the origin). I also want to call out here as an FYI and to avoid confusion that you may receive the SpatialAnchorManager.LocateAnchorsCompleted event before all of the SpatialAnchorManager.AnchorLocated events have completed.

robbbbbb commented 3 years ago

Hi @msftradford, thanks for working through all that.

I can confirm that the general problem as I reported was that I wasn't receiving SpatialAnchorManager.AnchorLocated. When this happened, I definitely didn't receive LocateAnchorsCompleted.

I have also seen examples of what you've observed: AnchorLocated is called, but GetPose() on the anchor returns Pose.identity and the "didn't find the anchor" message is printed. This identity pose issue is less common that the first. It sounds like you weren't getting the identity pose issue consistently yourself, but only occasionally?

I currently have a support ticket open with MS support in which one of your colleagues suggested upgrading my OS version from 20348.1018 to 20348.1432. This (along with 2.10.2 ASA SDK and 1.1.1 MR OpenXR package) seems to fix both issues, however there are now a lot of different variables which seem to have some impact on this:

As I say updating to the absolute latest of all of those:

Seems to produce a working build. What is checked in to the samples repo definitely doesn't. On the basis of your comment above you should be able to fix your own project by:

Do you guys have plans to update this repo itself with OpenXR-compatible examples? It would be really useful to have working code in this repo for that since it's now the recommended development environment in favour of WMR. So people following the standard HoloLens 2 "Get the Tools" documentation will get OpenXR, checkout the samples in this repo, and end up stuck. Having working OpenXR samples in this repo and being able to look at the various package versions (and perhaps release notes stating lower bounds on OS + OpenXR runtime versions) would be extremely useful in diagnosing SDK issues like this.

msftradford commented 3 years ago

@robbbbbb thanks so much for detailing this for us.

The GetPose() issue (#36539781) will be addressed in the next ASA SDK release.

Once we get to the bottom of this issue, we'll be sure to update our documentation to reflect the correct working versions of each module (#36808186).

We are working on the change (#36128208) to convert our Unity sample in this repo to use MROpenXR instead of WinXR and will let you know when it is released.

robbbbbb commented 3 years ago

Thanks @msftradford, it strongly sounds like the root cause of the original AnchorsLocated issue is OS-version related.

I don't have the time to test all combinations of different packages involved, but with the latest ASA SDK, MR OpenXR SDK and OpenXR plugin, I can definitely say that 20348.1432 does work and 20348.1018 does not.

This presents us a bit of a problem for knowing how and when to release our app with the latest updates, since we have no control over the OS version our users have installed.

Is it going to be feasible to update either ASA SDK or MR OpenXR SDK to fix the AnchorsLocated issue for earlier OS versions? If not could you guys at least investigate on which OS version this problem started (I assume it was present prior to 20348.1018, but how early?)

msftradford commented 3 years ago

Hey @robbbbbb we're working to root cause this (#36910921) so it is hard to say if we can provide a fix at the ASA or MROpenXR level yet. We'll keep this thread updated as we learn more. Thanks!

phamqduc commented 3 years ago

Hi @robbbbbb , we want to share with you further updates that ASA SDK 2.11.0 was released recently and it addresses the bug #36539781 and #36808186 mentioned above. We will let you know when updates on other issues are available. Thanks for your patience.

phamqduc commented 2 years ago

Closing the issue due to a lack of response. If you have other related questions, please kindly let us know or re-open the issue, @robbbbbb. Thank you.