Loosing focus when I minimise the application

[Morgane-Minsar]

Description

Opening the app after minimise it and move a bit, anchors found are misplaced.

Steps to reproduce the issue

I fork your repository to this one and I duplicated the scene AzureSpatialAnchorsCoarseRelocDemo to AdvancedAzureSpatialAnchorsCoarseRelocDemo as well as the script AzureSpatialAnchorsCoarseRelocDemoScript for AzureSpatialAnchorsCoarseRelocDemoScriptAdvanced I use in my new scene.

1. open the app and go to the scene AdvancedAzureSpatialAnchorsCoarseRelocDemo
2. place some anchors in front of you
3. when you have the watcher started (at the creation of a new anchor or at the step where you find nearby anchors), close the application without killing it
4. walk
5. open again the application : your cubes will still be in front of you instead of staying where you placed them

Expected behavior

At the opening of the application, cubes should be placed where I placed them.

Screenshots

Development information (please complete the following information)

• OS : macOS
• Development Platform\Version: Unity 2021.1.27f1 & Xcode 13.1
• Azure Spatial Anchors SDK version: 2.11.0

AR Device information (please complete the following information):

• Device: iPhone 12 pro
• OS: 15.0

[msftradford]

Hello @Morgane-Minsar, thanks for reaching out! It is important to distinguish between the device tracker and ASA. After your cloud anchors have been created/located (using the ASA SDK and service), they are anchored to your environment using ARFoundation's ARAnchors. At that point, the ASA SDK and service are no longer involved. The behavior of an ARAnchor is totally up to the device tracker and the ARFoundation layer. You can repro this exact behavior using ARFoundation's sample application in the "Simple AR" mode. I would recommend reaching out to the ARFoundation folks to determine the best way to handle this app switching behavior. Perhaps there is a way to reset the device tracker while keeping the anchors around. I'm going to go ahead and close this for now but do let us know what you learn. Thanks!

[tinanigro]

Hello @msftradford,

Your answer is technically correct, as ASA relies on the ARSession provided by ARCore / ARKit through ARFoundation. If the underlying ARSession isn't properly recovered between a minimize/maximize user operation, then it's no surprise ASA cannot properly work.

The scenario @Morgane-Minsar wrote happens all day every day. Everyone minimizes apps, sometimes just for a few seconds to check another app. AR-powered apps are no exceptions.

Let's consider the following scenario:

• User places an anchor (1) at location (A).
• User minimizes the app, then walks 3/5/100... meters.
• User re-opens the app at this new location (B).
• The previous anchor (1) is still there (because the underlying AR Session could not understand that the user has changed location, or at least ARFoundation is unaware).
• User adds another anchor (2) close to anchor (1).

Now, the user kills the app, and then they re-open it at location B. If the app uses Coarse Relocalization, both anchors 1 & 2 will be relocalized at location B, even though anchor 1 shouldn't. And if they open the app at location A, both anchors will be found, even though anchor 2 should not.

From a user's point of view, this is completely unacceptable. And if it's not acceptable for our users we cannot accept it either!

I understand this is not exactly an issue that originates from the ASA SDK, however as partners of Unity (and therefore the ARFoundation team), you should be the ones that collaborate with them to make sure such a regular scenario is properly supported.

At the very least, there should be some guidance or note about this scenario in the ASA documentation, to tell developers using your product that they may need to perform manual work to support that scenario. Ideally, you would reach out to the ARFoundation team, and try to investigate if there's a way for ASA to cancel/reset an existing ASA Session if the underlying ARSession is unable to properly recover (because of a change of location of otherwise).

This scenario is already documented on Apple's website for ARKit (Recover from Session Interruptions's section). I'm not knowledgeable enough about ARFoundation and its subsystems to confidently say that this information is surfaced from ARKit to ARFoundation, but either case, this should be a conversation between you and their team.

If we, as users of ASA SDK, must explain these things to the ARFoundation team, we will obviously have less impact than if you do!

Thanks,

[msftradford]

Hey @tinanigro, thanks very much for the details. I think that I understand the scenario and the request. Happy to work with you on how we can give your users better guidance while they are creating their experiences. Hopefully we can figure out how to improve our anchor relationships concept page (#37275020) and we can do some work to figure out how ARFoundation expects anchoring systems to handle tracking interruptions (#37275031).

It appears that you are taking advantage of anchor relationships when you perform your coarse relocalization queries. Because the anchor relationships (essentially poses between anchors) are established by the underlying platform that is tracking the device movement, the accuracy of the relationship is beholden to the accuracy of the tracker. There are a variety of ways the tracker can introduce error into the pose, including session interruptions, so it is best practice to direct the experience creators to visually verify the pose accuracy of the existing anchors in the scene before creating a new anchor that is intended to be connected to the existing anchors.

If anchors are in incorrect locations for whatever reason, the best practice would be to re-query the anchors before creating new anchors that will have relationships to the existing anchors. You can do this by resetting the ASA session with SpatialAnchorManager.ResetSessionAsync() and re-querying the anchors or by keeping the current ASA session and setting up a CloudSpatialAnchorWatcher with BypassCache set to true.

I'll also mention an alternative that might not be applicable to your experience but I want to call it out for other developers who might be following along. Instead of performing queries that leverage anchor relationships (LocateStrategy = AnyStrategy or LocateStrategy = Relationship), you can explicitly request that each anchor is matched visually using LocateStrategy = VisualInformation. That will make sure that device tracker interruption between anchor creations does not interfere with the accuracy of returned anchors.

[tomkrikorian]

Hi @msftradford, Sorry to get back to you after such a long time. We tried the BypassCache = true setting and it does improve the situation in our case.

And for the LocateStrategy = VisualInformation we noticed we actually are already using that setting since LocateStrategy = AnyStrategy when using a NearDeviceCriteria is basically LocateStrategy = VisualInformation according to your documentation : https://docs.microsoft.com/en-us/azure/spatial-anchors/concepts/anchor-locate-strategy#valid-combinations-of-locatestrategy-and-anchorlocatecriteria-properties

We were already doing a SpatialAnchorManager.ResetSessionAsync() before starting a new watcher when resuming the app so nothing to change on that front.

[phamqduc]

Hi @tomkrikorian, thanks for sharing updates and I'm glad that setting BypassCache improves the situation.

[msftradford]

Hey @tomkrikorian, great to hear that bypassing the cache it helping!

One thing that I should clear up here is that if you are using a AnchorLocateCriteria.Strategy of AnyStrategy in conjunction with AnchorLocateCriteria.NearDevice and there is only one anchor around you or you set the AnchorLocateCriteria.NearDevice.MaxResultCount to 1, then yes it is true that the anchor that is returned will have used VisualInformation. However, I think it is worth noting for other devs that if there are connected anchors nearby and the AnchorLocateCriteria.NearDevice.MaxResultCount is greater than 1, then you may be returned anchors that have a LocateStrategy of Relationship unless you specify the AnchorLocateCriteria.Strategy as VisualInformation.

@tomkrikorian, if that makes sense and checks out with what you're seeing, then we'll make a note to update that docs page to be more explicit.

[tomkrikorian]

Ho yes i understand better now. I think this should definitely be added to the docs. We did find anchors by relation as soon as the first anchor is discovered when using the AnyStrategy with NearDeviceCriteria. Now that we have set to VisualInformation it's not the case anymore.

[Varnauld]

Assigning to @pamistel for documentation update (37718188)