[Bug] ARFoundation 5.0.2 drift issue

[angecroft]

I discussed this issue on this thread, but it was closed from the author so I reopened it but for iOS. https://github.com/Unity-Technologies/arfoundation-samples/issues/1049#issuecomment-1401315855

Unity bug report case number IN-29813 - https://unity3d.atlassian.net/servicedesk/customer/portal/2/IN-29813

Describe the bug On iOS, a drift is very noticeable when the user moves their device, even slowly. When objects are in place they should not drift when the camera moves.

To Reproduce This is reproducible from the ARFoundation Samples 5.0.2. It has not been in any way modified other than opened in the Unity 2021.3.11f1, in order to illustrate our issue. You can build the project on iOS and test either "Simple AR", "Plane Detection" or in the case of the video files attached to this bug report, "Meshing > Normal Meshes".

Versions The drift is noticeable on iOS devices :

• iPhone 12 Pro (iOS 16.3)
• iPhone 14 Pro (iOS 16.1.2) It was observed with ARFoundation 5.0.2 with Unity 2021.3.11f1

Everything was working fine on an older version of ARFoundation (4.1.7) on Unity 2019.4.33.

I'd like to know if this issue is being considered or not because it's very blocking for now as I can record my work.

[therussmorris]

Interesting to see you re-post about this almost at the exact time I thought I'd come back and see if there was any progress made on the issue. I'd love to use AR Foundation 5, but seems it's still not stable enough to do so?

Has there been any progress on this @andyb-unity ?

[andyb-unity]

Hi @angecroft,

Thanks for opening a new issue for iOS. Your videos in your bug report clearly show the issue, so thanks for those as well!

I have a theory that this is actually a rendering-related regression. Tracking data should be correct (as our tracking code hasn't changed between versions), but at some point between Unity 2019.4 / ARF 4.1 and Unity 2021.3 / ARF 5.0, it seems possible that frame-perfect alignment of camera and tracking data was lost. In other words, the camera background appears to be one frame ahead of the AR content, causing a perceived "drift".

@therussmorris Am I correct that you think AR Foundation 5 is the source of the problem? ie, for the same device and same version of Unity, ARF 5 drifts but ARF 4.1 (and 4.2?) does not drift?

Thanks for bringing this to our attention and helping the investigation.

[andyb-unity]

We will check this at this as well, but if you get the chance please use AR Foundation / ARKit 5.0.3 or newer for your ARF 5 tests. The 5.0.3 patch has some significant bugfixes for other iOS camera issues: https://docs.unity3d.com/Packages/com.unity.xr.arkit@5.0/changelog/CHANGELOG.html

[therussmorris]

@andyb-unity Yeah, based on the my experience I've had looking in to it, that the issues are solely related to ARF 5 - as reverting back and testing with 4 has no problems at all.

Interesting that you think it could be related to frame synching the tracking and rendering - that does make a lot of sense!

Happy to help further and test anything you might be worth looking at. Cheers!

[andyb-unity]

Thanks @therussmorris. We believe we just reproduced the issue in XR Simulation as well, so I'm currently convinced this is indeed a frame sync issue and it affects all platforms. We've identified this as a high priority issue and will update this thread when we have more information about a potential fix.

[angecroft]

Hi @andyb-unity. I was just checking out if by chance you have any news on the potential fix? Thanks!

[andyb-unity]

Hi @angecroft at this point the key suspect in our investigation is the Input System's TrackedPoseDriver component. We are working internally to try to get this work scheduled with the Input System team, so not much to update yet.

[genereddick]

I realize you are working on this but I thought I would add that we are seeing this on iOS and with both ARF 4.n and 5.n.

Not drifty, running ARF 4.n2 and the previous ARSessionOrigin stack:

https://user-images.githubusercontent.com/1033793/230694034-2fec1fda-3aa1-4859-b7b4-34129bb10e70.MP4

And drifty running ARF 5.n and the new XROrigin stack

https://user-images.githubusercontent.com/1033793/230694054-da9aedea-1c6e-4b0d-a40f-119c7e25b677.MP4

[dganzella]

@andyb-unity I'm also seeing this on iPhone 12 Pro w/ iOS 16.4

[andyb-unity]

Thanks for the reports, everyone. We have enough evidence and have reproduced this issue on all platforms (ARCore, ARKit, and XR Simulation). Our internal partners have asked us for more information, and we are working to get this assigned to the right team with everything they need to resolve the issue. Not much to share yet but will continue to post updates here when we have them.

[dganzella]

If anyone needs a quick fix (don't we all need a quick fix from time to time?), just remove TrackedPoseDriver from the camera and add ARPoseDriver (even though its deprecated)

this workaround should be good enough until the official fix comes along

[genereddick]

It's been four months. Any update on this?

[andyb-unity]

Hey all, as you can see by the pull requests coming in that are starting to reference this issue, we are getting close! This investigation is still ongoing, and we have a team of 5 folks pitching in across AR Foundation, XR Interaction Toolkit, and Input System teams.

We're still not sure where or when all the changes will land, but in the meantime we're grateful for the workaround posted above by @dganzella. We will update this thread when we have more information.

[teemub]

Just wanted to say that we're very much looking forward to this fix! It has caused a bit of pain for us even if we have the workaround.

[micsanbr]

This also happens with AR Foundation 4.2.8 and not just 5.x. Those who report 4.2.8 as working correctly must be using the old AR Origin prefab and not the new "XR Origin (Mobile AR)" that can be created from the GameObject menu.

I just started working on a new AR app with 2022.3 LTS using AR Foundation 4.2.8 (included in the AR Template). After testing on Android I immediately noticed how laggy and unstable the tracking was.

Luckily I had an older AR Foundation project built with 2021.3 LTS, also using AR Foundation 4.2.8, which tracked very well. After trying to match every single project setting and scene setting -- what a waste of time -- I've decided to replace TrackedPoseDriver with ARPoseDriver as recommended by @dganzella I was very reluctant to change this first because the AR Pose Driver shows a deprecation warning text.

@andyb-unity It's amazing to see this kind of bug in a "stable" release. Nobody tested AR Foundation 4.2.8 with an actual project on a device before releasing it? Are the Unity devs perhaps testing new releases using the old prefabs? This happens on Android (Google Pixel 6) and it has been reported on iOS (iPhone 14 Pro) as the original poster wrote. Without this workaround AR Foundation 4.2.8 is basically useless. Is it going to to take months before we see a fix, possibly also in the 4.2.8 branch and not only for 5.x ? Does this workaround cause any problem (missing features or incompatibility with the ARPoseDriver)?

[micsanbr]

There's another semi-issue introduced with the InputSystem dependency,

Here's how to reproduce it.

Create a new project using the AR template in 2022 LTS. Create a new BLANK scene. Crease a new XR Origin -- select GameObject > XR > XR Origin (Mobile AR). Select the XR Origin and disable the TrackedPoseDriver component. Select the XR Origin and turn it into a prefab. Now create a new scene and import the XR Origin prefab. Look at the TrackedPoseDriver and notice that it lost the PositionInput and RotationInput bindings simply because the TrackedPoseDriver component was disabled.

Shouldn't the bindings persist? The bindings persist in my tests if the TrackedPoseDriver remains enabled. Is this a bug related to the InputSystem package or to the AR Foundation package?

[andyb-unity]

@micsan13br some answers for you:

Shouldn't the bindings persist?

My expectation would be that yes they should persist.

Is this a bug related to the InputSystem package or to the AR Foundation package?

Definitely not AR Foundation. We don't own TrackedPoseDriver, which is why related issues have been more challenging to debug. Hard to say if this is an Input System issue or a Unity Engine issue. Please file a bug so Unity QA can track down the correct team to fix it.

Nobody tested AR Foundation 4.2.8 with an actual project on a device before releasing it? Are the Unity devs perhaps testing new releases using the old prefabs?

We test every AR Foundation release against the AR Foundation samples branch associated with that version. In this case you're right actually-- our 4.2 branch of AR Foundation samples uses the old prefabs.

Is it going to to take months before we see a fix

Possibly. We have had one of our engineers assigned to this issue full-time for several weeks now, and we feel like we are getting close, but we can't make any promises on delivery dates for this fix.

possibly also in the 4.2.8 branch and not only for 5.x ?

We will definitely patch all affected versions of whatever is causing this issue. At this time we are still not 100% sure about which combination of AR Foundation, Input System, and Unity Engine will require a patch. But once we have the correct fix we will definitely communicate this information.

I was very reluctant to change this first because the AR Pose Driver shows a deprecation warning text. Does this workaround cause any problem (missing features or incompatibility with the ARPoseDriver)?

By rules of Semantic Versioning, we cannot remove ARPoseDriver except in a major version release of AR Foundation. So AR Foundation 4 and 5 will always include ARPoseDriver. We haven't started development of AR Foundation 6 yet, but we will not remove ARPoseDriver until this investigation is complete. There are no known issues with ARPoseDriver, and you can suppress the deprecation warnings with a pragma warning disable if you wish.

Depending on the outcome of our investigation, we are also considering un-deprecating ARPoseDriver. At this time we still think fixing the interaction with the Input System is possible but again we'll share more when we have the answers.

Hope that helps, and thanks for your engagement on this issue. I agree that the AR Foundation 5.0 release is tarnished until this issue is fixed, and our work continues.

[genereddick]

re: "Does this workaround cause any problem (missing features or incompatibility with the ARPoseDriver)?"

We use an extensive number of ARF features have been using the ARPoseDriver since we first tested this fix and have not seen any issues. Can't say 100% there are none of course.

[micsanbr]

Thank you @andyb-unity for the exhaustive answers. Thanks @genereddick as well for confirming that ARPoseDriver still works well.

Here's one final thought, which might be of course completely off, but still ... after swapping the TrackedPoseDriver with ARPoseDriver I also haven't noticed anything strange, which makes me wonder: is it really a good idea to replace ARPoseDriver with another Component that in theory offers a bit more flexibility (customizable bindings) that 99% users will never need but introduces a cross-dependency with the InputSystem package? Ok, it's also broken but that can happen.

I know that the InputSystem is the future for input management but people are still working with older LTS releases (for ex. 2021 LTS because 2022 LTS was released two weeks ago). These users are using assets and packages (some from Unity) that still make use of the old InputManager.
I'm aware that both Input methods can coexist but still the Editor itself aggressively tries to disable the old input manager in certain situations. This happened to me last week while checking the samples for another package.

[andyb-unity]

Hi all, we have an update on this issue.

We have confirmed that the source of this issue is solely in Unity Editor code, so the fix will come in the form of patches to all affected Editor versions. To access this fix, you will need to upgrade your Editor version to one of forthcoming patch releases (or newer), and you will not need to change your AR Foundation version.

We have made a pull request to the 2023.2 alpha branch of the Editor, and once that is approved we will be able to backport to older Editor versions. Once those backport pull requests are approved and merged in, we can share the version numbers of the Editor releases that will contain this fix.

Thanks for everyone's engagement in reporting this issue. Looking forward to getting this fix out to everyone soon.

[andyb-unity]

Status update: our pull request for this issue is approved and merged, and will land in Unity 2023.2.0a23.

Backports for the following Editor versions are in progress:

• 2023.1
• 2022.3
• 2021.3

We'll post those patch versions as well once we're merged in.

[OktawianWasilewski]

on unity 2023.1 and ARF 5.1.0-pre.4 have rly bad simple image tracking its same bug ? now im trying downgrade my app and ARF to older version. When image is detecting then move device slowly/fast and have ultra drifts and lose tracking. I cant use this ARF 5x versions its unstable. You can download from this repo sample image tracking and its not stable.

[angecroft]

That's awesome news @andyb-unity! Thank you for the support

[micsanbr]

@andyb-unity Thanks a lot. Please keep us posted about when this lands in LTS branch.

[andyb-unity]

All of our backports for this issue have been merged. Here are all the Editor version numbers that contain the fix:

• 2023.2.0a23 (reposted from above)
• 2023.1.5f1
• 2022.3.6f1
• 2021.3.29f1

[genereddick]

Great, thanks for the update

[andyb-unity]

A minor correction to my note above:

There is no such version 2023.2.0a23. 2023.2 Beta 1 was released last week, and it contains this fix.

[ragavendranbala]

The issue persists in 2022.3.6f1 with MRTK3, HoloLens 2. AR foundation version is 5.0.2 MRTK3 version 3.0.0-pre.17

[andyb-unity]

Closing this issue as fixed. @ragavendranbala if your issue persists, it is for some other reason. Microsoft owns and maintains their AR Foundation implementations for HoloLens; I recommend that you ask in their Mixed Reality forums for further guidance.