How to add a ReferenceImage to MutableRuntimeReferenceImageLibrary from Resources?

[eugeneloza]

Hi all!

I'm having a problem detecting a ReferenceImage loaded by Resources.Load<Texture2D>(name) into a MutableRuntimeReferenceImageLibrary.

More specifically:

I have a QRCode image (copied from arfoundation-samples repository). If I add it to a ReferenceImageLibrary asset created in Editor it is detected and tracked correctly.

However, when I add the very same image through Resources.Load<Texture2D>("QRCode"); - the image isn't tracked. The texture is read/write enabled. The image is not null, JobHandle.IsCompleted eventually reports true (also tried JobHandle.Complete();). There seems to be no related errors/warnings/notes in the log. But eventually ARTrackedImageManager.trackedImagesChanged is never called.

In another project with the same Unity/ARFoundation version I get the Texture2D from XRCpuImage.ConvertAsync and the image is added runtime and is detected correctly in a runtime-created MutableRuntimeReferenceImageLibrary. So this part is also working correctly. There doesn't seem to be any significant difference in projects settings.

The minimal code that reproduces the issue:

Texture2D image = Resources.Load<Texture2D>("QRCode");
trackedImageManager.referenceLibrary = trackedImageManager.CreateRuntimeLibrary();
MutableRuntimeReferenceImageLibraryExtensions.ScheduleAddImageJob(
(MutableRuntimeReferenceImageLibrary)trackedImageManager.referenceLibrary,
image, "QRCode", 0.15f).Complete();

Maybe it expects some specific texture format? Or does some implicit color or geometry transformations of the texture?

At some point I had an error log about unsupported ETC4 format, but it was fixed by explicitly specifying RGBA 32 bit or RGB 24 bit in the texture import settings. Also I tried explicitly converting (by dst.SetPixels(src.GetPixels()); dst.Apply(); and dst.SetPixels32(src.GetPixels32()); dst.Apply();) the texture into TextureFormat.RGB24 that is working successfully for XRCpuImage in the AR camera program, but still no luck.

Unity 2020.1.6f1 and 2020.1.5f1 ARFoundation 4.1.0-preview.7 SM-A405FN, Android 10

Might be related to https://github.com/Unity-Technologies/arfoundation-samples/issues/536

[tdmowrer]

Please see https://github.com/Unity-Technologies/arfoundation-samples/issues/586#issuecomment-693112008 for an explanation.

TL;DR: The image is detected but not tracked until you "explore" the space, e.g., by moving your device around a little.

[eugeneloza]

Thanks a lot for the comment. I've just read the post (but not yet the whole topic - I'll study it more carefully tomorrow).

The image is detected

The problem is that the image is not detected at all. I.e.:

void OnEnable()
{
  trackedImageManager.trackedImagesChanged += OnTrackedImagesChanged;
}
...
private static void OnTrackedImagesChanged(ARTrackedImagesChangedEventArgs eventArgs)
{
    Debug.Log("OnTrackedImagesChanged"); //<-- this log is never displayed if image(s) were loaded from Resources
}

Moving around a lot for over a minute doesn't help. When in AR Camera app I'm detecting an image I just took a photo of, indeed, I have like 0.5-2 seconds delay and indeed I need to move the phone around a bit to start tracking the image properly. But when I load the texture through Resources.Load the image is not detected for over 1-2 minutes, never for any test of dozens I've made today and yesterday; moving around, changing looking at the image on the monitor and printed on paper a few times, rotating, changing tilt. OnTrackedImagesChanged is never called and never seen in the logs.

However, indeed I've noticed before, that a prebuilt library "catches" the image almost immediately, while dynamically loaded ones require some noticeable time, which is, though not critical, but might be an unwelcome delay. So, I'm now thinking of using a prebuilt library - which is while not the most versatile solution, but looks like it has only "pros" - quicker image detection and more wide devices support.

Thank you for the link again, I think it almost persuaded me to rework the image detection architecture into using a prebuilt library tomorrow :)

[tdmowrer]

The image tracking sample in this repo has a "Dynamic Library" script that will add images from a list when a button is pressed. I removed the QRCode from the prebuilt XRReferenceImageLibrary and added it to the list of dynamic images. It works, just with the behavior described by https://github.com/Unity-Technologies/arfoundation-samples/issues/586#issuecomment-693112008. I also tried adding it from a Resources.Load<Texture2D> and that also worked.

Prebuilt libraries generally perform better, but a couple quick points on the code you posted (if you continue to use dynamic libraries):

Texture2D image = Resources.Load<Texture2D>("QRCode");
trackedImageManager.referenceLibrary = trackedImageManager.CreateRuntimeLibrary();
MutableRuntimeReferenceImageLibraryExtensions.ScheduleAddImageJob(
(MutableRuntimeReferenceImageLibrary)trackedImageManager.referenceLibrary,
image, "QRCode", 0.15f).Complete();

Is quite verbose. Extension methods extend an existing type, so you can just call

library.ScheduleAddImageJob(image, "QRCode", 0.15f);

The reason this method schedules a job is that it can be quite expensive to add an image (ARCore's documentation states that it can take 20-30 ms, an entire frame). If you call Complete you are forcing it to complete before continuing, which negates any advantage of using the job system (though I understand you may have just been calling Complete to debug this particular case).

And finally, you should check that the runtime library is indeed mutable (or your explicit cast will throw). I suggest something like this:

if (trackedImageManager.CreateRuntimeLibrary() is MutableRuntimeReferenceImageLibrary library)
{
    var image = Resources.Load<Texture2D>("QRCode");
    library.ScheduleAddImageJob(image, "QRCode", 0.15f);
    trackedImageManager.referenceLibrary = library;
}

[eugeneloza]

I also tried adding it from a Resources.Load and that also worked.

Thank you for the test! Hmm... Then there's something wrong on my side.

Is quite verbose ... check that the runtime library is indeed mutable ... to debug this particular case

Yeah exactly, that's a cropped down to viable minimum part of a much larger project, made to work "quick and dirty" in conditions similar to those it should work in - without verbose checks, loading images with parameters from game data, queuing images, waiting for scheduled job to finish, replacing library if needed, etc. Of course loading a single image can be made much simpler and cleaner.

Thank you very much for your help!

And as I've already decided to rework everything into a prebuilt library, then this issue seems no longer relevant. I'll close it.