[Bug] [Android] [ImageTracking] Image tracking doesn't get updated after initial detection

[RobAEG]

Image tracking doesn't get updated after initial detection When using a MutableImageLibrary, after adding an image to the mutable image library and pointing the device camera to the tracking image, it gets detected. But subsequently it will not detect any changes in movement. This mainly happens with smaller tracking images, around the size of 3 by 3 centimeters. The tracking images themselves are QR codes. Larger tracking images, also QR codes, suffer the same problem from time to time but not as frequently.

In my code, I use ARFoundation's ARTrackedImagesChangedEventArgs struct to handle the detection, updated and removal of tracked images. The ARTrackedImagesChangedEventArgs "added" event gets received almost as soon as I have the image in view, which is great. But subsequently no ARTrackedImagesChangedEventArgs "updated" events are received unless I move the camera up and down and around the tracking image. Once the tracking image has been "updated" in this way, tracking seems to be fairly stable.

What could be the cause for this delay in "updated" events? On physically larger tracking image, also of QR codes, a slight delay sometimes happens as well, but not as frequently or annoyingly as the one for smaller tracking images. Changing the expected physical size to the physical size of the tracking image when adding it to the mutable image library doesn't seem to help alleviate this issue.

On iOS devices tracking proceeds as expected, with the "added" event being followed up with the "updated" event almost immediately even when tracking smaller tracking images. Any help or insight for this issue is appreciated!

Additional information Builds were tested on these devices:

Motorola One Macro, Android Version 9 One Plus 5, Android Version 10

Builds were made with the latest versions of the AR Foundation & AR Core packages, and build on Unity version 2019.4.6f1

[DominicdeGraaf]

Important to note that it generally takes 10-20 seconds before the updated event is properly called on Android. It usually takes less than 1 second on iOS.

[tdmowrer]

Important to note that it generally takes 10-20 seconds before the updated event is properly called on Android. It usually takes less than 1 second on iOS.

That's not what I see in this samples repo. Adding a new image takes ~1 second or less to detect it. I imagine it depends on the quality of the image (from ARCore's standpoint). There's a command line tool you can run that will give you a "score" for how suitable the image is for ARCore.

[DominicdeGraaf]

I just cloned this sample repo in Unity 2019.4.8f1 and added two of our own images to the DynamicLibrary.cs example script in the "BasicImageTracking" example scene.

I've also checked these two images(link and link) in the command line tool you suggested, and both return a score of 100.

When I test these images on my monitor, they are generally recognized within 1-2 seconds, when I print them out it takes ~10 seconds on average. And I have to move my phone around slowly, if I don't then they won't get recognized at all.

• When I add one of our own images to the ReferenceImageLibrary (as static image), they get detected straight away.
• When I remove the Unity logo example from the static ReferenceImageLibrary and add it as dynamic image, it again, takes ~10 second on average before it gets recognized.
• Images on a monitor get recognized a LOT faster than printed versions, most notably when using dynamic images. (monitor 1-2 seconds, printed 10-20 seconds).

To reproduce:

• Remove one of the static images from the library and add it to the DynamicLibrary.cs example script. Print it out, and notice how it takes a much longer time before it is recognized and Android (ios is fine!)
• Add THIS image to the static library and print it out and notice it is recognized almost instantly.
• Add THIS image to the dynamic library and print it out and notice it takes 10-20 seconds before it is recognized.

I hope we will be able to figure out why this is not working correctly!

[tdmowrer]

I believe the issues discussed in this thread are largely artifacts of the way ARCore performs image tracking. I will try to summarize here, but you might get a more detailed answer from the ARCore folks by posting on https://github.com/google-ar/arcore-unity-sdk/issues

ARCore treats images in the same way it treats the rest of the environment, that is, its detection and tracking is based on feature points. This works well for unique images, like a painting or photograph, but QR codes all look alike in that they have very regular features.

I have to move my phone around slowly

Even though ARCore might give your images a score of 100, because its tracking algorithm is still based on feature points, it requires that you move the device (or at least benefits greatly from it). By contrast, most QR code scanning apps require you to hold still so it can see the image clearly, so we've all been "trained" not to move our phones when scanning a QR code. This is not how ARCore's image tracking works, so holding the device still is actually a worst-case for its detection algorithm.

If all you want to do is scan a QR code, feeding the raw image pixels to a bar code reader might be a more appropriate solution. That has been discussed a few times on this issues page, e.g., https://github.com/Unity-Technologies/arfoundation-samples/issues/489

This mainly happens with smaller tracking images, around the size of 3 by 3 centimeters.

With regards to size, ARCore doesn't really care how big the image is, but it does matter how much of it fills its field of view. Smaller images mean the user must move closer to the image, and this may cause autofocus to struggle with very close objects.

When I test these images on my monitor, they are generally recognized within 1-2 seconds, when I print them out it takes ~10 seconds on average.

ARCore's image detection is based largely on contrast, so I would imagine a backlit LCD screen displaying a black and white QR code has optimal contrast, while a printed QR code would have much less.

I hope that helps provide some context. I don't believe there is anything more we can do in ARFoundation, since it is just an abstraction of ARCore in this case.

[DominicdeGraaf]

Hi Tim,

I'd like to stress, that adding these QR codes to the static library, they are recognized almost instantly. Tracking images that are added to the mutable library take a lot longer to be recognized. Please do try it out with printed versions of the example images provided in the samples AND with the QR

If you really need to, I can prove the correct tracking of QRcodes in a static library and the not correct recognition of the sample images provided in a mutable (dynamic) library. But that will take some effort on my part on making a video. You can easily test it yourself though by testing with the images I provided in my previous post. Also, if you download the AR Scanner app (https://play.google.com/store/apps/details?id=com.stickylock.arscanner), I've developed this app using AR Core instead of AR Foundation and there the AR tracking works a lot better (we only use QR codes for recognition and they work great). See the app in working here(https://youtu.be/woC03f8qSGk), I know it looks like that QRcode isn't the tracking image, but it definately is.

I really do hope I can make you believe me in saying that there is a problem regarding tracking in AR Foundation with AR Core. We've completely rewritten our AR apps to AR Foundation because we believe that this will provide in way more stable AR performance. However for now, this bug is killing our release.

[mdurand42]

Can your reproduce this delay with the images included our AR Foundation sample? Small QR codes are especially challenging for ARCore according to their engineers.

You might also try adding your QRCode image to the AR Foundation sample and checking if you see the same behavior as you are seeing in your app.

[DominicdeGraaf]

Hi guys,

Let forget for a while that I am using QR codes as tracking images.

The bug that I am reporting, while using a clone of the sample project, is that the image recognition of images in a MutableRuntimeReferenceImageLibrary takes a lot longer than images that are already added to the static reference library.

To reproduce:

• Print of the three images from the sample project.

• create a build of the basicimagetracking example scene.

• notice very quick image recognition.

• remove all three images from the static reference library and add them to the dynamiclibrary.cs example script. This will add them to the MutableRuntimeReferenceImageLibrary

• again create a build of the basicimagetracking sample scene.

• in de build press the “load images” onGUI button.

• now the example images are much more difficult to get recognized.

I am seeing these results consistently using only the example project and its example images.

[tdmowrer]

Oh, I know what the problem is.

When you build an image library at build time, if the source images are JPG or PNG, then we feed those directly to the ARCore image database generation tool. At runtime, you can either supply either an array of bytes representing your image, or you can use a an extension method that accepts Texture2Ds. By default, a Texture2D's import settings alter the image so that the dimensions are rounded to the nearest power of 2. The rafflesia photo, for example, is a JPEG at 1024x681. However, when imported as a Texture2D into Unity, it gets rounded to 1024x512:

You can even see how it looks "squashed" in the preview.

So if you add an image dynamically in this way, the image you end up adding to the image library is not the same as the image you are trying to find, which is probably why ARCore struggles to find it in this case. The images in our sample one.png and two.png happen to be 512x512 already, so they do not suffer from this problem.

If you want to dynamically add an image to a reference image library as a Texture2D, then you can change the import settings to not alter the image dimensions. This, however, is less optimal if you intend to ever send this texture to the GPU for rendering.

[DominicdeGraaf]

I will definatly try if this is the issue I am having, however, I have also tried this with already square images (like the Unity logo example image), which gave the same results. I’ll report back to you when I know more.

[DominicdeGraaf]

Hey guys, I've played around with some of the settings, and it seems that disabling the mipmapping speeds it up a bit, I have indeed disabled Non-Power of 2.

So what I did next: I took 3 of my QR images, gave them the exact same inport settings. I added 2 of them to the static reference library, and one to the dynamic library. I filmed the results:

https://www.dropbox.com/s/4b0byrqin68i1ls/IMG_3468.MOV?dl=0

Notice the 3 QR codes in the video, the most left and most right are added to the static library, the middle one to the dynamic library. I tried it multiple times (as seen in the video), you'll see that the static images are recognized much, much faster than the one in the dynamic library.

To give it a proper test, I also changed the images around, added the left image to the dynamic library and the middle and right one to the static library: https://www.dropbox.com/s/3ma6nksedp09au5/IMG_3469.MOV?dl=0

Any other idea's? See the texture import settings, (No Android Overrides):

[DominicdeGraaf]

Any new idea's guys?

[tdmowrer]

I will be looking into this more deeply this week.

[tdmowrer]

I just cloned this sample repo in Unity 2019.4.8f1 and added two of our own images to the DynamicLibrary.cs example script in the "BasicImageTracking" example scene.

I've also checked these two images(link and link) in the command line tool you suggested, and both return a score of 100.

When I test these images on my monitor, they are generally recognized within 1-2 seconds, when I print them out it takes ~10 seconds on average. And I have to move my phone around slowly, if I don't then they won't get recognized at all.

• When I add one of our own images to the ReferenceImageLibrary (as static image), they get detected straight away.
• When I remove the Unity logo example from the static ReferenceImageLibrary and add it as dynamic image, it again, takes ~10 second on average before it gets recognized.
• Images on a monitor get recognized a LOT faster than printed versions, most notably when using dynamic images. (monitor 1-2 seconds, printed 10-20 seconds).

To reproduce:

• Remove one of the static images from the library and add it to the DynamicLibrary.cs example script. Print it out, and notice how it takes a much longer time before it is recognized and Android (ios is fine!)
• Add THIS image to the static library and print it out and notice it is recognized almost instantly.
• Add THIS image to the dynamic library and print it out and notice it takes 10-20 seconds before it is recognized.

I hope we will be able to figure out why this is not working correctly!

I have followed your repro steps using your QR codes stoel and apollo.

• I printed the images on paper and placed them side-by-side on my desk.
• I created an XRReferenceImageLibrary containing both images.
  • Each image is recognized quickly (hard to measure exactly, but feels like less than 1 second; very snappy).
• I then removed "apollo" from the reference image library and added it dynamically on button press.

This is where things get interesting. If the images are relatively large (16 x 16 cm), there is no noticeable difference.

However, when I shrank the QR codes to 8x8 cm (probably closer to your use case), I am able to reproduce some of what you are seeing. However, it's not that it takes a long time to detect the image -- it is detected in less than half a second (even as quickly as 0.15s) -- but its tracking state remains TrackingState.None until you "explore" the space a bit (i.e., move your phone around). In ARCore, this maps to AR_AUGMENTED_IMAGE_TRACKING_METHOD_NOT_TRACKING. During this time, the position and extents are meaningless (usually it's just all zeros). Most of our samples hide the GameObject until the tracking state is something other than None, so this would give the appearance that the image takes a long time to be detected.

On the other hand, the image that was part of the original, statically generated reference library ("stoel") is detected and tracked without having to move the device.

I've confirmed this difference at the native C level as well; ARFoundation is not misreporting the tracking state. Dynamically added images are detected almost immediately and updated pretty frequently, but they will remain at AR_AUGMENTED_IMAGE_TRACKING_METHOD_NOT_TRACKING until you explore the space a bit, while images from the prebuilt library do not seem to require this.

Note all this assumes I'm holding my phone perfectly still. As soon as I move a little, it seems to pick up enough features to move to full tracking.

In conclusion, while I acknowledge there is a difference in behavior between static and dynamic reference images, that difference seems to be an implementation detail of ARCore. You can still determine that a particular QR code was detected very quickly, you just won't have good tracking data until the user moves their device a bit. The tracking state of an ARTrackedImage is available from its trackingState property.

[DominicdeGraaf]

Hi Tim,

Thanks for investigating! However, I have one more item to add to the investigation!

We have a live Android app, called “AR Scanner” this has been built in Unity 2017, without the AR Foundations package, but instead with the AR Core Unity Git repo that had to be used back than.

This live app detects the dynamically added images almost instantly and also updates the tracking straight away.

However, AR Foundation adds a lot better and stable tracking features we though, so we wanted to rebuild the app from the ground up to AR Foundation. For now, this has been a waste of time, because the app works can’t be updated for Android in its current state because of the very bad dynamic image tracking functionality.

You can get the free app here: https://play.google.com/store/apps/details?id=com.stickylock.arscanner&hl=en_US

Do you have any idea why the older AR Core project works so much better?

[tdmowrer]

Do you have any idea why the older AR Core project works so much better?

I would say that "better" is subjective, or at least use case dependent, here. In some sense, QR code detection is kind of a worse-case scenario. ARCore's image tracking, like every other feature, depends on feature point extraction from the environment, unlike a bog-standard QR code reader which is just looking for a specific kind of image.

"Google Play Services for AR" (ARCore) has gone through several updates since then. Back in 2017, your app may have still used a developer preview. I've noticed significant improvements in the image tracking feature in each update, but while the API remains backwards compatible, the algorithms have changed and won't necessarily produce the same behavior.

So I don't know specifically why you see different behavior, but it is not entirely surprising, given nearly 3 years of development and updates.

[StickylockMax]

Hi Tim,

Thanks for investigating! However, I have one more item to add to the investigation!

We have a live Android app, called “AR Scanner” this has been built in Unity 2017, without the AR Foundations package, but instead with the AR Core Unity Git repo that had to be used back than.

This live app detects the dynamically added images almost instantly and also updates the tracking straight away.

However, AR Foundation adds a lot better and stable tracking features we though, so we wanted to rebuild the app from the ground up to AR Foundation. For now, this has been a waste of time, because the app works can’t be updated for Android in its current state because of the very bad dynamic image tracking functionality.

You can get the free app here: https://play.google.com/store/apps/details?id=com.stickylock.arscanner&hl=en_US

Do you have any idea why the older AR Core project works so much better?

To add to Dominics reply, I've added a few QR codes that we are using.

[DominicdeGraaf]

So you know who I can address best on the AR Core side of things than? If you say its in the native C level, I suppose I have to get in contact with them?

[tdmowrer]

So you know who I can address best on the AR Core side of things than? If you say its in the native C level, I suppose I have to get in contact with them?

https://github.com/google-ar/arcore-android-sdk/issues is their issues page.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[DominicdeGraaf]

Update: We posted this on the AR Core github.

They confirmed that the dynamic library should update the same as the static library and will look into it.

Our post: https://github.com/google-ar/arcore-android-sdk/issues/1137

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Hless]

I too am seeing this issue still. I can hardly believe that the issue is at the C level of the ARCore library. Should it not be a widely known issue to all developers using ARCore if that was the case? I have also used another library before ARFoundation (not unity) pinned on exactly the same ARCore version, that did not have this delay.

I could try and investigate if this occurs by directly handling the ARCore NDK libraries in an example app? Would that help to further investigate the issue?

[DominicdeGraaf]

Someone from the AR Core Github is also looking into it, perhaps we can him. (google-ar/arcore-android-sdk#1137).

It seems nobody from Unity will be looking further into it though, so that guy seems our best shot.

[Hless]

I understand that nobody from Unity is looking into it, since the issue is thought to be in ARCore. It will take a long time to fix if that is the case, but since I doubt that it is (perhaps wrongly), I'm offering to write up an example app using the ARCore version in the latest stable ARFoundation version.

If my doubts are correct, we might greatly improve time to fix, since the example can be used as a reference. If not, we are sure that it must be fixed in a future ARCore version.

added: @tdmowrer please understand that I aim to move past this problem as quickly as possible, we are seeing a degradation in service to our Android users since switching our AR solution over to ARFoundation. This is painful, and I want to help to find the culprit

[DominicdeGraaf]

@Hless I'm sure it will increase our chances of getting our products/services stable sooner greatly. It seems that you have a good idea, so I can only hope you find something!

[Chroust007]

I found that there is no problem with image detection. It is recognized immediately (cca 3sec). Unfortunately, Mutable library ignores the physical dimensions of the image. That's why it takes a long time for tracking to start. I searched the entire AR Core code and the AR Foundation extension, but didn't figure out how to force the dimensions of images, during convert into the AR library. Do you have some ideas, haw to do this?

[tdmowrer]

The dimensions are not ignored. How did the arrive at that conclusion?

[Chroust007]

The mutable images size hs always different. When you start tracking, the images is detecting with size (0,0) and tracking state "Added". Then it take a long time for detecting size. Its the 20 sec delay between detect and set tracking state to update. Always with little difference size, especially when you are zooming phone.

[tdmowrer]

This is the behavior of ARCore. Please read my earlier comment: https://github.com/Unity-Technologies/arfoundation-samples/issues/586#issuecomment-693112008, specifically:

...it's not that it takes a long time to detect the image -- it is detected in less than half a second (even as quickly as 0.15s) -- but its tracking state remains TrackingState.None until you "explore" the space a bit (i.e., move your phone around). In ARCore, this maps to AR_AUGMENTED_IMAGE_TRACKING_METHOD_NOT_TRACKING. During this time, the position and extents are meaningless (usually it's just all zeros).

[Chroust007]

Yes, but the none tracking state (i wrote wrong Added) is probably due to missing dimension in ARCore runtime database. Haw is possible, that there is always detecting different size when you are zooming phone? It could be exactly same size, as you defined at ScheduleAddImageJob. This would also explain the different behavior - the worse tracking then images from static library.

[tdmowrer]

Well, I certainly have egg on my face. There is indeed a regression in 4.0+ which does not supply the correct physical dimensions to ARCore when an image is added at runtime. In previous versions, there was one function which handled both cases; now there are two different functions depending on whether the physical dimensions are specified or not, and we only ever call the one that does not take physical dimensions.

This has been fixed in 4.1.0-preview.10 released yesterday. It will also be backported to 4.0.x verified.

[DominicdeGraaf]

Hi Tim,

I've been trying to update ARCore to 4.1.0 preview 10, unfortunately it gives me a lot of Gradle build errors.

"UnityEditor.BuildPlayerWindow+BuildMethodException: 78 errors"

One of which is:

"Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0."

Any idea?

[Hless]

@tdmowrer Ah, great, will check it out when I can.

@DominicdeGraaf try using a gradle version < 6.0 for building with Android. There is some guidance on how to swap gradle versions for Unity here: https://answers.unity.com/questions/1490992/change-version-of-gradle-to-46.html

Edit: Note, to quickly test you could also export the Android project and change the gradle version by opening the project in Android studio and changing it there.

[DominicdeGraaf]

Changing gradle fixed one of the many errors...

[tdmowrer]

Please see https://developers.google.com/ar/develop/unity/android-11-build

TL;DR 2020.1+ should work out of the box. 2019.4 or earlier requires you to upgrade the version of gradle.

@DominicdeGraaf

Changing gradle fixed one of the many errors...

Can you provide a full list of errors?

[DominicdeGraaf]

Thanks Tim,

i’ll try it out tomorroe.

[Chroust007]

Amazing, Good news! Thank you, Tim. Now It works like a charm.

[tdmowrer]

Amazing, Good news! Thank you, Tim. Now It works like a charm.

Great! Apologies that took so long to track down.

[DominicdeGraaf]

This was it, also the reason for us why the tracking took ~10 to 20 seconds. Now almost instantly.

[Hless]

Yup, can also confirm, tracks as fast as to what we were used to before :)

Thanks @Chroust007 for pointing in the right direction.

[DominicdeGraaf]

PS, I do want to let everybody working with QR codes as tracking images with the knowledge that tracking of QR codes works great.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.