Tracking Andy objects drifts

[Zumbalamambo]

Thank you for the wonderful code. I tried to use your code to track the Andy objects in HelloAR example. When I wear the headset and view, the andy objects are drifting. Any possible fix for the same?

[ChristophGeske]

When the tracking fails it can happen that the ARCore camera and all the planes where the andy object is positioned on starts drifting away. You can try to move more slowly and make sure that the environment is feature rich and well lit to avoid tracking errors. I often experience drifting in the first seconds after starting the app since ARCore needs some time to stabilize and the ARCore Camera jumps around. When you touch the touchpad on the GearVR the position is reset and changes to the position caused by initial drift can be reset.

Does that make things clearer? If not feel free to ask agein.

[Zumbalamambo]

Have you tried to remove ARCoreBackgroundRenderer and render it stereoscopically with black screen? When you do so, the andy objects are drifting. Is there any possible fix for the same?

[ChristophGeske]

Have you tried to remove ARCoreBackgroundRenderer and render it stereoscopically with black screen? When you do so, the andy objects are drifting. Is there any possible fix for the same?

I would wonder if the drifting has something to do with enabling or disabling those scripts. As far as I know the ARCoreBackgroundRenderer is only for displaying the camera image so disabling it or switching to a stereoscopic camera should not change the tracking. The andy object is positioned on the planes which are generated by the point cloud. If the tracking fails the point cloud returns bad data, the planes start drifting and since the andy object is positioned on the planes the andy object drifts away.

If you wanna fix it you need to stop trusting that ARCore is always correct and not fix the andy object to the planes. Instead you would first check the position data of the planes and move the andy object if you trust the data. If you don't trust the data for example tracking fails because the room is to dark or bright for the camera you rely on other sensors or don't move the andy object at all. This way you prevent the andy object from following a drifting plane. If tracking starts working again you can let the andy object follow the plane again.

From your explanation I think you experience tracking issues and there is no simple fix for it I suppose.

[Zumbalamambo]

@ChristophGeske sorry for the late reply and thank you for the wonderful explanation. I have noticed that the tracking is perfect but the visualization makes it appear as if the Andy object is drifting with the camera movement. Could there be any solution for the same?

[ChristophGeske]

You mean you have issues when using the Andy object which is part of the HalloAR app and you try to visualize them using VR mode? I know that this is somehow tricky and doesn't work out of the box to visualize objects in VR that are places in AR. I am not sure how to solve this issue since I haven't experimented in this direction much but you can try the following if it makes sense to you.

When you place an object in the real world by touching your screen in the HalloAR app the Andy is placed at a position somewhere in front of your camera. The position the Andy object is placed on is in relation to a feature point in the real world. Now switching to VR mode it is possible that the perceived distance changes and movements in VR might make the Andy object appear to behave strange. I would try to bring the position of the Andy which is based on the real world feature point into the virtual world. Let me try to explain how I would try to implement this. First: You would place the Andy object somewhere based on a real world feature point. Second: You now measure the distance between VR head camera and the andy object in the virtual world. Next: You use this distance and you create a new Andy object in the virtual world at the distance away from your head you just measured. This should now look as if the Andy positioned on the feature point and the andy in the virtual world based on the distance to your head are at the exact same position. But the point where the andy is standing on is no longer based on the real world but based on the virtual world because we set it in relation to the virtual camera. Next: You can delete the andy that is based on the real world feature point so you are left with the andy that's positioned in the virtual world in relation to your VR camera. Now when the tracking is ok the Andy should appear to stay where it is in the virtual world and any disturbances in the real world would not affect the position of your Andy anymore. Only when the camera loses tracking completely everything would start to drift away.

That's just an idea and I am not sure if it works.

[Zumbalamambo]

Could you please suggest how I can measure the distance between the VR head camera and the andy object in the virtual world? Is there any transformation to do so?

[ChristophGeske]

When I think about it you don't really need to calculate the distance between the head and the andy there is an easier way. If you create the andy it is positioned in the real world and also displayed in the virtual world. When you move andy stays positioned relative to the real world but is displayed in the virtual world which I think might lead to the drift you experience. So what you can try is to record the position the andy is displayed in the virtual world when you first place it and then create a new andy at this exact position in the virtual world.

So the steps you can try would be:

1. Place the andy object which is normally done by touching the screen. The andy is now positioned in the real world and displayed in the virtual world.
2. Get the position where the andy is displaced in the virtual world by using Vector3 andyPos = andyObject.transform.position ; Where andyObject is the andy Object in your scene. To get andys position you need to access the andy GameObject via your script. That might work with a get Component GameObject function or the andy is stored as a trackable object by ARCore so you might find it in the list of trackable objects. There are possibly other ways to access a gameObject in a scene.
3. Now create a new Andy or a cube for testing.
4. Place a cube at the stored position with cubeObject.transform.position = andyPos ;
5. Check if the cube appears to stay where it should and if the andy keeps drifting. If they are booth drifting my idea was wrong and something else is causing the drift.

As I said I am not sure if that solves your issues but that's what I would try first to see if that's the problem. Also it's not so easy to help you this way since I can only guess what is going on in your code. You could publish your code or create a new issue as a feature request describing in detail what you are trying to create so maybe someone has an idea how to implement it and can write a script for you.

[Zumbalamambo]

I have tried doing so but unfortunately, it isn't working out. it still moves. is there any other fix for the same? :(

[ChristophGeske]

Next I would print all the relevant data and try to understand what is drifting and what might causing it. You could simply use Debug.Log("Andy x Position: " + andyObject.transform.position.x) and put the line in the Update method to get a reading of the andy position each frame and compare it to the position of the camera Debug.Log("Camera x Position: " + camera.transform.position.x) and other objects in you scene.

The only time when I see drift is at the beginning of the app which is caused by ARCore searching for feature points and not having a good understanding of the real world yet but this gets better after a few seconds. I also see drift when there are not enough feature points in the environment.

But as you describe it you have a different kind of drift which is somehow related to the andy object. So as of now I have no other idea and would look at the data and find out what's drifting exactly and why it does so.

[Zumbalamambo]

@ChristophGeske sure, let me try it. Could it also be because it is being tracked but the visualization with black background makes it look like its drifting?

[ChristophGeske]

I do not know if there is a connection. The background visualization is not what I would think about first when I experiencing drift. Looking at the real data should bring you to the right answer quicker than guessing what the reason might be. If the drift is affected by the background visualization than the data should tell you when changing the background settings.

[ChristophGeske]

I should have mentioned that recording data when the app is running on the phone is more complicated. I know of two ways.

1. Via usb cable. This is complicated to set up but allows you to quickly change thing in the code and see whats causing errors. For that you need the phone developer mode and the GearVR developer mode active. The app needs to be build in developer mode. You might need to set up the android development bridge correctly and make sure the device is connected. You use the Debug Logs for printing the data. You switch the profiler from editor to your android device.

2. Using the StreamWriter and write to a text file on you phone. // For Recording/Debugging public StreamWriter writer; public List positionDataList = new List(2000); int counter = 0; // For Recording/Debugging */ private void PositionRecordingDebugging() { counter++; if (counter == 1890) { for (int i = 0; i < positionDataList.Count; i++) { writer = new StreamWriter(Application.persistentDataPath + "/posLog.txt", true); writer.WriteLine(positionDataList[i]); writer.Close(); } } else { if (counter > 1900) { return; } else { if (50 < counter && counter < 1890) { positionDataList.Add(" Posx: " + currentState.transform.position.x ); } } } } This method takes longer but you don't need a teather. You can then view and process the data in Excel for example.