How to bind reference image with SimulatedTrackedImage

[lookoutking]

As stated in the manual:

To optionally bind a simulated tracked image to your reference image library, set the Image field of its Simulated Tracked Image component to reference a texture asset that is also used in the reference image library.

I've tried to bind a simulated tracked image with one in the reference image library by assigning the same texture asset. However, the reference image is null in the simulated tracked image during the simulation.

OS: Windows 10 Unity: 2021.3.6f1 ARFoundation: 5.0.3

Thanks in advance for any help.

[andyb-unity]

Are you using the same serialized image library that is assigned to the Serialized Library field of your AR Tracked Image Manager? I just double checked and this worked as expected for me.

Note that XR Simulation does not currently support mutable reference image libraries-- the reference image needs to be saved into the library before you enter Play mode.

[lookoutking]

I found that the Serialized Library field becomes null right after entering Play mode. Maybe this is the problem.

The tracked image library is assigned before entering Play mode But the field suddenly becomes None after entering Play mode

Do you have any idea what happens? Thanks for the help.

[therussmorris]

Also experiencing the same issue here. Keen to know if you managed a way to work around this?

[therussmorris]

It seems like it is by design that you can't currently differentiate between different SimulatedTrackedImage markers in the simulated environments?

To clarify, I'm using AR Foundation 5.0.3.

Any AR Tracked Image asset created after scanning a SimulatedTrackedImage doesn't acquire any data from the reference image library, basically it's just creating an empty object - so the reference image data can't be used in editor to differentiate between one target and another. It seems it's just a way to know if something has been scanned, but nothing specific. Again, seems by design, but would be good to get a confirmation on this as I haven't seen anything that confirms this in documentation? @andyb-unity is that right?

As mentioned above, the serialised library does indeed become null, but again this seems like it's by design as it's only used as a reference to assign to the subsystem at runtime. That's my understanding of it anyway.

If the SimulatedTrackedImage workflow is how I've described it above, I'd be keen to know if this is something that is planned to be expanded on in the future?

Alternatively, I guess doing a bespoke solution might be the best way forward? Raycasting to a specific object in the scene and using data assigned to it to trigger events that would occur after scanning specific markers - I'm sure there are downfalls to this, but could solve the problem of replicating a feature set of spawning objects related to the scanned marker that seems to be missing?

[lookoutking]

It seems like it is by design that you can't currently differentiate between different SimulatedTrackedImage markers in the simulated environments?

To clarify, I'm using AR Foundation 5.0.3.

Any AR Tracked Image asset created after scanning a SimulatedTrackedImage doesn't acquire any data from the reference image library, basically it's just creating an empty object - so the reference image data can't be used in editor to differentiate between one target and another. It seems it's just a way to know if something has been scanned, but nothing specific. Again, seems by design, but would be good to get a confirmation on this as I haven't seen anything that confirms this in documentation? @andyb-unity is that right?

As mentioned above, the serialised library does indeed become null, but again this seems like it's by design as it's only used as a reference to assign to the subsystem at runtime. That's my understanding of it anyway.

If the SimulatedTrackedImage workflow is how I've described it above, I'd be keen to know if this is something that is planned to be expanded on in the future?

Alternatively, I guess doing a bespoke solution might be the best way forward? Raycasting to a specific object in the scene and using data assigned to it to trigger events that would occur after scanning specific markers - I'm sure there are downfalls to this, but could solve the problem of replicating a feature set of spawning objects related to the scanned marker that seems to be missing?

I actually had a similar workaround as the alternative solution you described, just want to try this new feature (SimulatedTrackedImage) to improve my current workflow. I didn't find a way to figure the problem finally and agree with your understanding on it. Hope this could be confirmed.

[andyb-unity]

so the reference image data can't be used in editor to differentiate between one target and another. It seems it's just a way to know if something has been scanned, but nothing specific.

This is not the case. See the documentation for the SimulatedTrackedImage component as quoted below. (Was this documentation too difficult to find? I wrote these docs and feedback is certainly welcome.)

XR Simulation can detect and track all simulated tracked images in an environment, even if you have not included their textures in your reference image library. To optionally bind a simulated tracked image to your reference image library, set the Image field of its Simulated Tracked Image component to reference a texture asset that is also used in the reference image library. When XR Simulation detects images that are not part of the reference image library, the corresponding ARTrackedImage trackables will not contain a fully initialized referenceImage. Instead, the guid property of the referenceImage is set to zero, and its texture is set to null.

To make sure we're clear, here is an example from a scratch scene on my machine:

This scene has 5 images, and I'm logging their reference images using this simple script I wrote (ignore the warnings in my console):

using System;
using UnityEngine;
using UnityEngine.XR.ARFoundation;

[RequireComponent(typeof(ARTrackedImageManager))]
public class ImageLogger : MonoBehaviour
{
    ARTrackedImageManager m_ImageManager;

    void Awake()
    {
        m_ImageManager = GetComponent<ARTrackedImageManager>();
        m_ImageManager.trackedImagesChanged += OnTrackedImagesChanged;
    }

    static void OnTrackedImagesChanged(ARTrackedImagesChangedEventArgs args)
    {
        foreach (var image in args.added)
        {
            Debug.Log(image.referenceImage);
        }
    }
}

I can inspect the Simulated Tracked Image components in my simulation environment by going to Window > XR > AR Foundation > XR Environment, then selecting Edit Environment from the Create/Edit Environment dropdown.

When I inspect the environment, I can see that my Simulated Tracked Images are all set up with Image properties that match what is displayed in my scene. This is strongly recommended. Simulated Image Tracking relies solely on the fields of the Simulated Tracked Image component; it does not actually do any optical texture recognition.

To optionally bind a simulated tracked image to your reference image library, set the Image field of its Simulated Tracked Image component to reference a texture asset that is also used in the reference image library.

In my case, 2 of these Images are in the reference image library set on my AR Tracked Image Manager before I entered play mode. I can see these reference images, 'QRCode' and 'Rafflesia', and their GUIDs, etc, in my logs. 3 of the images in my scene are NOT in the reference image library, so the image tracking subsystem doesn't know anything about them. To make XR Simulation easier to use, we decided to trigger an image detection event with an uninitialized reference image in this case. This is the case you describe where "something was detected", but hopefully now it's clearer how you can set up your simulation environment to give you more specific information about the reference image.

(PS: Yes the AR Tracked Image Manager sets the serialized image library to null when it starts up. I personally find that to be a confusing choice but yes it's by design. The serialized library is converted to a RuntimeReferenceImageLibrary before subsystem start.)

[ROBYER1]

What threw me off was that the image markers need to be in the Simulated Environment and don't work if they exist within the actual scene, the documentation could make that clearer.

[lookoutking]

In my case, 2 of these Images are in the reference image library set on my AR Tracked Image Manager before I entered play mode. I can see these reference images, 'QRCode' and 'Rafflesia', and their GUIDs, etc, in my logs. 3 of the images in my scene are NOT in the reference image library, so the image tracking subsystem doesn't know anything about them. To make XR Simulation easier to use, we decided to trigger an image detection event with an uninitialized reference image in this case. This is the case you describe where "something was detected", but hopefully now it's clearer how you can set up your simulation environment to give you more specific information about the reference image.

(PS: Yes the AR Tracked Image Manager sets the serialized image library to null when it starts up. I personally find that to be a confusing choice but yes it's by design. The serialized library is converted to a RuntimeReferenceImageLibrary before subsystem start.)

Thank you for the reply. I think the document is clear and easy-to-find. But I can not correctly build up what you demos. I already assigned the Image field in SimulatedTrackedImage. It seems that the ImageLogger does not print non-zero GUID as expected, also the image.name is a random-assigned name.

(Note that I use ARFoundation 5.0.3, Unity 2021.3.6f1 the same as the original post)

[andyb-unity]

And is the earth texture also in the TestReferenceLibrary that you have assigned to your image manager? If so, please file a bug as I can't tell what else could cause this issue: https://unity3d.com/unity/qa/bug-reporting

[lookoutking]

And is the earth texture also in the TestReferenceLibrary that you have assigned to your image manager? If so, please file a bug as I can't tell what else could cause this issue: https://unity3d.com/unity/qa/bug-reporting

Thanks, I reported and this is the bug case id: IN-30915.

[andyb-unity]

Hi @lookoutking,

I took a look at your project. The issue is that you need to set Keep Texture at Runtime to true for your reference images in the reference image library. Note the tooltip:

If enabled, the texture will be available in the Player. Otherwise, the texture will be null in the Player.

The Simulation Image Tracking provider gives you texture guid information by comparing the texture GUID in the simulated tracked image component to textures in your runtime reference image library. If the runtime reference image library contains null textures, XR Simulation will report null textures as well.

It has occurred to me that there is a limitation here, where XR Simulation might motivate you to choose one setting for Keep Texture at Runtime, but you might want a different setting in your builds. I've made a note to see if we can address this limitation in a future version of AR Foundation.

[lookoutking]

Hi @lookoutking,

I took a look at your project. The issue is that you need to set Keep Texture at Runtime to true for your reference images in the reference image library. Note the tooltip:

If enabled, the texture will be available in the Player. Otherwise, the texture will be null in the Player.

The Simulation Image Tracking provider gives you texture guid information by comparing the texture GUID in the simulated tracked image component to textures in your runtime reference image library. If the runtime reference image library contains null textures, XR Simulation will report null textures as well.

It has occurred to me that there is a limitation here, where XR Simulation might motivate you to choose one setting for Keep Texture at Runtime, but you might want a different setting in your builds. I've made a note to see if we can address this limitation in a future version of AR Foundation.

Thanks for the helps! It works fine.