[Bug] StartCoroutine Methods don't go well when used in scenes using ARFoundation features

[Sightwalker]

Description I experienced 2 behaviours that (I believe) came from the same problem. On the project I'm working on the system downloads an AssetBundle using a StartCoroutine method when a target image is recognized. While the AssetBundle is downloading, there's a Slider component working as a loading bar.

-The first behavior was that sometimes when the target got recognized and started downloading the AssetBundle, the screen would freeze. Along with the freeze, some "glitched squares" appeared. Originally was just one, but when I added canvas elements, more of them started to appear. (see screenshot below). These glitched squares seem to be the app trying to display the screen with the ARCamera trying to show the contents seen by the device camera, since the glitched parts change colors that match the ambient. It became more clear when the canvas elements were added, because they are still functional (even the ones that appear on the glitched squares)

I worked around it by changing an if condition that checked if the trackingState was None to check if it was Limited. Then the glitch would only occur when the tracked image was lost during the download process. Still not satisfied, I added to the if condition a bool checking if the object was finally downloaded. Then finally the bug stopped happening.

Screenshot: (just after trying to read the Monalisa image target)

-The second behavior is more subtle. It happens when you can track multiple image targets. Whenever there is a target that already had it's content downloaded and it's active, when the app finds another target that wasn't loaded yet and starts downloading the AssetBundle content, the lighting of the scene appears to spin. Now I don't know for sure if it's a lighting problem or if the objects are moving inside the "Unity Scene" (but they aren't moving in the actual scene as they're "glued" to the image target).

For this behavior, I tried to set Light Estimation on the AR Camera Manager to "none" but the lighting continued to spin. I found out that this "lighting spinning" also happened when playing a video that's still being downloaded.

I already found out a workaround for the first behavior so I hope it helps if someone has a similar problem, but I'm still trying to figure out what I can do about the second one. I didn't test it on iOS to see any of this would happen.

Using:

• Devices: Samsung Galaxy Tab S4, Samsung Galaxy S9 and Xiaomi Mi9
• OS: Android 10
• Unity 2020.1.3f1
• ARFoundation 4.0.8 and/or 4.0.9

[tdmowrer]

That sounds very strange and it is hard for me to imagine how downloading an AssetBundle in a coroutine is connected to an image's tracking state. I'm also not sure how to attempt to reproduce this.

It would be helpful if you could post some code so I can get a better idea of what you are doing.

[Sightwalker]

I had to work around a code that alredy existed when I got hired, so please keep that in mind

It uses the OnEnable and OnDisable methods to assign the events and all that. For each tracked image in the eventArgs that was added or update, it calls this function:

    void SpawnGO(ARTrackedImage trackedImage, string mode)
    {
        for (int i = 0; i < arraylenght; i++)
        {
            if (referenceImageNames[i] == trackedImage.referenceImage.name)
            {
                string metadataType = mdAssetBundle.metadata[i];
                metadataType = metadataType.Substring(0, 6);

                //if the argument of the new update is .add, it instantiates obj at location and adds to the list at appropriate index
                if (mode == "add")
                {
                    StartCoroutine(GetAssetBundle(mdAssetBundle.metadata[i], trackedImage.referenceImage.name, trackedImage.transform, i));
                }
                else if (mode == "update")
                {
                    //if the argument of the new update is .update, it simply mantains object spawned and updates its location if target has moved
                    spawnedObjects[i].transform.position = trackedImage.transform.position;
                    spawnedObjects[i].SetActive(true);
                }

                //De-spawns the object if the tracked image has been fully compromised
                if (isLoaded[i] && trackedImage.trackingState == TrackingState.Limited)
                {
                    spawnedObjects[i].SetActive(false);
                }
            }
        }
    }

And then the GetAssetBundle function goes like this:

    IEnumerator GetAssetBundle(string fullUrl, string targetName, Transform target, int index)
    {
        UnityWebRequest www = new UnityWebRequest(fullUrl);
        DownloadHandlerAssetBundle handler = new DownloadHandlerAssetBundle(www.url, 0);
        www.downloadHandler = handler;
        var operation = www.SendWebRequest();

        if (www.isNetworkError || www.isHttpError)
        {
            Debug.Log(www.error);
        }
        else
        {
            while (!operation.isDone)
            {
                var downloadDataProgress = operation.progress * 100;
                yield return null;
            }

            // Extracts AssetBundle
            AssetBundle bundle = handler.assetBundle;
            names = bundle.GetAllAssetNames();
            var prefab = bundle.LoadAsset<GameObject>(names[0]);
            var instantiatedObject = Instantiate(prefab, new Vector3(target.transform.position.x, target.transform.position.y, target.transform.position.z), Quaternion.Euler(0, 0, 0));
            instantiatedObject.transform.eulerAngles = target.transform.eulerAngles;
            instantiatedObject.SetActive(true);
            spawnedObjects[index] = instantiatedObject;
            isLoaded[index] = true;
            bundle.Unload(false);
        }
    }

About the second behavior that I mentioned in the first post, I am now sure that it isn't the Light component that is spinning while a download coroutine is in action, but, in fact, all the objects in the scene (that aren't tied to an image target) spin while those coroutines are in progress.

[tdmowrer]

That seems to work fine for me. I even added an extra 5 second loading period in your while (!operation.isDone) loop to simulate a longer download time. The one thing that I did notice is that

else if (mode == "update")
{
    //if the argument of the new update is .update, it simply mantains object spawned and updates its location if target has moved
    spawnedObjects[i].transform.position = trackedImage.transform.position;
    spawnedObjects[i].SetActive(true);
}

does not perform any null checks, and presumably, spawnedObjects[i] will be null until the asset bundle is downloaded and instantiated. Meanwhile, the image is likely to get updated (it was for me; I had to add null checks here).

That doesn't explain why all your content is spinning, but perhaps that exception interferes with logic elsewhere in your app?

[tdmowrer]

Something else that occurred to me: you said you had to change the logic which checked whether the tracking state was Limited to None. I assume you mean this:

//De-spawns the object if the tracked image has been fully compromised
if (isLoaded[i] && trackedImage.trackingState == TrackingState.Limited)
{
    spawnedObjects[i].SetActive(false);
}

In my attempt to reproduce this, I assumed that spawnedObjects and isLoaded are parallel arrays. In that case, you should never even get to the trackedImage.trackingState == TrackingState.Limited check while the coroutine is running, right? Because isLoaded is not set to true until the end.

[Sightwalker]

Oh. Right! I'm very sorry! The bool isLoaded was my workaround for the bug when I was trying to make the project work with trackedImage.trackingState == TrackingState.None To replicate the bug you need to remove the check for isLoaded[i]. The if statement should be looking like this:

if (trackedImage.trackingState == TrackingState.None)
{
      spawnedObjects[i].SetActive(false);
}

Now about this:

That seems to work fine for me. I even added an extra 5 second loading period in your while (!operation.isDone) loop to simulate a longer download time. The one thing that I did notice is that

does not perform any null checks, and presumably, spawnedObjects[i] will be null until the asset bundle is downloaded and instantiated. Meanwhile, the image is likely to get updated (it was for me; I had to add null checks here).

That doesn't explain why all your content is spinning, but perhaps that exception interferes with logic elsewhere in your app?

If it still doesn't work if you remove the bool check, maybe it should be a problem with the ARTrackedImageManager being instantiated at the start of the scene instead of already being there. Maybe it's some problem with mutable or changed libraries during runtime

[tdmowrer]

That's a lot of maybes :)

If I remove the isLoaded check, then I get a bunch of NullReferenceExceptions while the asset bundle is being loaded, which is what I would expect to see, but the rendering is not affected and objects do not spin around.

Since affecting that logic fixed your problem, my best guess is this is related to uncaught exceptions or logic on the spawned objects (since I don't know what they do) that breaks when you hit this code path. Any chance you can pair this down to a minimal repro project and submit it here?

Edit: Do you have any relevant logcat (adb logcat -s Unity) output that may point to the problem?

[Sightwalker]

That's a lot of maybes :)

If I remove the isLoaded check, then I get a bunch of NullReferenceExceptions while the asset bundle is being loaded, which is what I would expect to see, but the rendering is not affected and objects do not spin around.

Since affecting that logic fixed your problem, my best guess is this is related to uncaught exceptions or logic on the spawned objects (since I don't know what they do) that breaks when you hit this code path. Any chance you can pair this down to a minimal repro project and submit it here?

Thanks a lot for your attention (and yeah, I know that's a lot of maybes, sorry about that)!

I'll have to check with the company what I can do about this. If they give me an okay I'll sure do it.

Edit: Do you have any relevant logcat (adb logcat -s Unity) output that may point to the problem?

Sadly, I don't :(

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.