Can a tracked image be removed from a mutable image library?

[Britvich]

After dynamically adding a tracked image using ScheduleAddImageJob to a MutableRuntimeReferenceImageLibrary, is there a way to remove that image from the library, so that it is no longer tracked?

The docs say:

You should never Destroy a trackable component or its GameObject directly. For trackables that support manual removal, its manager will provide a method to remove it. For example, to remove an anchor, you would call RemoveAnchor on the ARAnchorManager.

Is there something like a RemoveImage for a MutableRuntimeReferenceImageLibrary?

[tdmowrer]

No, you cannot remove a reference image once it is added to a library (some SDKs do not allow us to do this). What is the use case for doing so?

The documentation you quoted refers to trackables, a different concept than reference images. A reference image is an image to look for in the environment. Once one has been found, the ARTrackedImageManager generates an ARTrackedImage, which is a trackable. The ARTrackedImageManager does not support removing tracked images.

[Britvich]

Use case: I want to dynamically look for Snickers bars and Captain Crunch boxes, then after one of them is found, stop looking for one or both of them, and dynamically change the set of looked for images.

Work around: I suppose anytime a reference image is removed (no longer to be looked for), I could just rebuild the entire list of current reference images.

Desired: I wish the API had something like a RemoveImage call to make it simpler. I also wish I didn't have to use ScheduleAddImageJob and handle all the job stuff myself. I would prefer just calling AddImage and let you handle all the gory details behind-the-scenes. ;)

[tdmowrer]

Use case: I want to dynamically look for Snickers bars and Captain Crunch boxes, then after one of them is found, stop looking for one or both of them, and dynamically change the set of looked for images.

Work around: I suppose anytime a reference image is removed (no longer to be looked for), I could just rebuild the entire list of current reference images.

You could also just ignore detected images you no longer care about.

Desired: I wish the API had something like a RemoveImage call to make it simpler.

Yes, that would be simpler; however, the underlying SDK has to provide this functionality. ARCore, for instance, has no way to remove reference images from an "augmented image database".

I also wish I didn't have to use ScheduleAddImageJob and handle all the job stuff myself. I would prefer just calling AddImage and let you handle all the gory details behind-the-scenes. ;)

What "job stuff" are you handling? You don't have to do anything other than call ScheduleAddImageJob unless you really care about knowing when the job completes.

It is called Schedule AddImage Job rather than just "AddImage" to let you know that the work it does is asynchronous, and it provides a JobHandle in case you want to query for completion or schedule additional work when that job completes. However, you can just call ScheduleAddImageJob and move on; you don't need to handle anything.

Also note that if you want to add multiple images to the database, the jobs don't have to depend on each other (i.e., you don't have to pass the job handle from one to the next).

[Britvich]

Thanks for the response. I'm an old developer and have never seen an API I didn't hate. ;)

I understand some SDKs don't currently support removing reference images. My point is to hide that detail inside your API, where after a RemoveImage call, the API would either remove it from the reference image set (if the underlying SDK supports that), or just stop reporting detections of that image (if the SDK can't actually remove the image from the set).

Good APIs keep things at the highest level possible, hiding from developers the underlying implementation details.

With ScheduleAddImageJob, don't I have to wait for it to complete before the ARTrackedImageManager referenceLibrary is set and enabled? I would still prefer an AddImage call, that produces something like a trackedImagesAdded event on completion, to make it more symmetrical with the rest of the API and also hide the underlying details of the use of the job system.

Imagine at some point in the future (or it might already be) where the underlying SDK itself asynchronously handles adding images and you (as the API implementer) don't even want to use Unity's job system. Since you've already exposed the use of Unity's job system at the API level, it's hard to no longer use it. The API user (developer) just wants to add/remove images to/from the set and doesn't care how that actually happens.

Thanks again for your ideas and I just wanted to let your know how helpful the AR Foundation has been to my work. I do appreciate all your hard work.

[tinanigro]

Hello AR explorers :)

Not having an API for removing an image from a mutable image library because one platform doesn't support it would make sense, only if this principle was applied globally in ARFoundation.

But in ARFoundation, it is exactly the opposite. ARFoundation is here to help developers leverage AR features that are common to most platforms while letting them benefit from more advanced capabilities. Here's a list of a few platform-specific capabilities in ARFoundation:

• setting a muted runtime library (oh the irony)
• tracking moving images
• automatically guessing the size of the tracked images
• the amount of maximum moving images to track

And here I'm only talking about Image Tracking in ARFoundation. There are so many platform-specific features available in ARFoundation, like depth/body occlusion, or object tracking with ARKit. EnvironmentProbes have manual/precise positionning capabilities with ARKit, and limited/automated positionning with ARCore: this difference of capabilities is also well handled in ARFoundation, which is super nice.

These platform-dependant capabilities are specified in the subsystem descriptor by the platform-specific subsystem implementation. This is exactly the way it should be, and it's great to see ARFoundation helping developers get the best of both worlds (common feature-set + pushing the limits on some platforms).

Here are a few suggestions and one potential bug report.

ARFoundation gets a ScheduleRemoveImageJob() method

Available on some platforms.

• The XRImageTrackingSubsystemDescriptor adds a new boolean property similar to supportsMutableLibrary, such as supportsRemovalOfImageInMutableLibrary (like XREnvironmentProbeSubsystemDescription.supportsRemovalOfManual).
• This property throws NotSupportedException("This subsystem does not track removing images from the mutable library"); when it's set on unsupported platforms *like XRImageTrackingSubsystem.maxNumberOfMovingImages which throws throw new NotSupportedException("This subsystem does not track moving images.");.

Documentation Update

The ARFoundation team updates the documentation to specify mutable image libraries are only partially mutable, since you cannot properly remove images from that library after adding them.

Potential bug

Our scenario is to be able to track image A, then image A+B+C, then remove image B from tracking. It is currently not possible. We also tried to track image A, destroy the ARTrackedImageManager, re-create it, re-configure its mutable image library, and adding the image to the mutable image library once again. What we end up with is only partially working: we properly receive the trackedImagesChanged callback, but the name of the ARImage and its texture are gone (probably badly cleaned native objects when destroying theARTrackedImageManager), even though the event itself returns accurate positions and rotations. We are currently investigating thoroughly this issue with a sample project and will submit a ticket if confirmed.

[tdmowrer]

Thanks for you comments, folks. All good suggestions and feedback.

@Britvich

Good APIs keep things at the highest level possible, hiding from developers the underlying implementation details.

I agree that good APIs make simple things simple (which means keeping things high level and hiding implementation details), but they also allow a path for more advanced usage and don't restrict those who want to interact with the implementation details.

With ScheduleAddImageJob, don't I have to wait for it to complete before the ARTrackedImageManager referenceLibrary is set and enabled?

No, you don't have to do anything. It's a fire-and-forget method if you don't care about job completion.

I would still prefer an AddImage call, that produces something like a trackedImagesAdded event on completion, to make it more symmetrical with the rest of the API and also hide the underlying details of the use of the job system.

In Unity, JobHandles are the consistent and expected way of interacting with the Unity job system. Not many things in ARFoundation use the job system, so perhaps it feels like a departure from ARFoundation's API surface, but certainly not Unity in general. While an event may be more consistent with the rest of the existing ARFoundation API, do you actually want that? Your original comment on this was

I would prefer just calling AddImage and let you handle all the gory details behind-the-scenes

which, if you ignore the return value, is exactly what ScheduleAddImagJob gives you.

@ThomasNigro

The XRImageTrackingSubsystemDescriptor adds a new boolean property similar to supportsMutableLibrary, such as supportsRemovalOfImageInMutableLibrary (like XREnvironmentProbeSubsystemDescription.supportsRemovalOfManual).

That is one possibility; we are considering something along these lines.

We also tried to track image A, destroy the ARTrackedImageManager, re-create it, re-configure its mutable image library, and adding the image to the mutable image library once again. What we end up with is only partially working: we properly receive the trackedImagesChanged callback, but the name of the ARImage and its texture are gone (probably badly cleaned native objects when destroying theARTrackedImageManager), even though the event itself returns accurate positions and rotations. We are currently investigating thoroughly this issue with a sample project and will submit a ticket if confirmed.

That sounds like a bug; please submit a bug report and post the case number here if you are able to produce such a sample project.

[eugeneloza]

Hi all! Just to drop my 5 cents in :)

My usecase: I'm developing an app that would allow users to check if the given image is performing well as a trackable image. I.e. the user takes a photo of the image, the app adds it to the reference image library and it immediately tries to detect it; then the user may try different angles, lighting conditions, etc. If the image is good, the user saves the image with metadata, if not, then... I'll ask the user to restart the app before taking another photo :) so that the old photo would not "get in the way" of working with a new one, which may be of the same object. This is an "internal" app and we can "afford" asking users to restart it every time. I'll also try a couple of workarounds that will reinit/recreate TrackableImageManager during runtime. But of course I'd gladly use the opportunity to simply remove the image from the reference library after it was used and saved or discarded.

By the way, thanks a lot - AR Foundation is a real jewel, very cool and easy to work with!

[tdmowrer]

Hi all! Just to drop my 5 cents in :)

My usecase: I'm developing an app that would allow users to check if the given image is performing well as a trackable image. I.e. the user takes a photo of the image, the app adds it to the reference image library and it immediately tries to detect it; then the user may try different angles, lighting conditions, etc. If the image is good, the user saves the image with metadata, if not, then... I'll ask the user to restart the app before taking another photo :) so that the old photo would not "get in the way" of working with a new one, which may be of the same object. This is an "internal" app and we can "afford" asking users to restart it every time. I'll also try a couple of workarounds that will reinit/recreate TrackableImageManager during runtime. But of course I'd gladly use the opportunity to simply remove the image from the reference library after it was used and saved or discarded.

By the way, thanks a lot - AR Foundation is a real jewel, very cool and easy to work with!

You don't need the ability to remove images from a reference image library in order to achieve this. Why not just create a new reference image library for the test image, set that to be the ARTrackedImageManager's referenceLibrary, then delete the test reference library if the image is bad?

[eugeneloza]

@tdmowrer thanks a lot! It worked like a charm!

[Blackclaws]

With ScheduleAddImageJob, don't I have to wait for it to complete before the ARTrackedImageManager referenceLibrary is set and enabled?

No, you don't have to do anything. It's a fire-and-forget method if you don't care about job completion.

Actually you have to make sure that you do not destroy the NativeMemory that you had to pass into that method before its actually finished.

Maybe I'm also misunderstand something about the job system here, but ScheduleAddImageJob states clearly that:

// The must be valid until this job completes. The caller is responsible for managing its memory

So if I want to reduce the memory usage of my application I have to make sure to wait for job completion in order to then clean up the NativeArray I had to pass in.

It would be much easier if the API surface just took any of the image formats that Texture2D can load and handled all the Allocation/Deallocation behind the scenes.

------------- Scratch everything above this line

I just found out that there are indeed Extension methods to the mutable library that provide this API surface. My bad.

[Hless]

@eugeneloza Trying the same thing with ARFoundation 4.0.2 on iOS. But for some reason the system keeps tracking previously added trackables. I'm unsure how to 'delete' a previous reference library. Haven't been able to find a method for it in in the documentation. I'm curious how you got rid of the old trackables in your implementation.

[eugeneloza]

@Hless hi!

I'm curious how you got rid of the old trackables in your implementation.

By replacing the old library with a new one, that contains only one image - that was enough for the usecase. I'll try to chop down everything to a minimal example. Internal variables used:

using Unity.Jobs;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
...
public ARTrackedImageManager arTrackedImageManager;
RuntimeReferenceImageLibrary newLibrary;
JobHandle jobHandle;

this is actually creating the new library and adding the image to it:

newLibrary = arTrackedImageManager.CreateRuntimeLibrary();
if (newLibrary is MutableRuntimeReferenceImageLibrary mutableLibrary)
{
    jobHandle = MutableRuntimeReferenceImageLibraryExtensions.ScheduleAddImageJob(mutableLibrary, imageTexture, imageName, 1);
}
else
{
   //log something about impossible to add images and give up
}

finally in Update() checking every frame if the job is completed:

if (jobHandle.IsCompleted)
{
    arTrackedImageManager.referenceLibrary = newLibrary;
    arTrackedImageManager.enabled = true;
}

Yes, obviously every image that was in the previous referenceLibrary is lost. So if you want to keep some of them, you'll need to add them to the new library together with the new images.

[Hless]

@eugeneloza Thank you for the elaborate example! I had almost the same thing implemented, however the old trackables still kept floating around even after reassigning a new library. I'm not sure what the reason is, perhaps its an ARKit issue, or it could be due to the fact I assigned a new library with exactly the same texture as before? (Sometimes I'm loading a different scene, but with the same trackables in the new library).

Anyway, I managed to fix it by calling Reset on the ARSession before entering a different scene/assigning a new library.

Example code:

var arSession = GetComponent<ARSession>();
arSession.Reset();

Edit: To specify my problem further: I think it's correct that the system tracks an identical texture with the same trackableId. However, for my implementation I was using _ARTrackedImage.referenceImage.name as a way to link gameObjects to the trackable. Upon assigning a new library the referenceImage got unset, instead of replaced with the referenceImage in the new library. To me this seems like a bug

[tdmowrer]

Couple quick points to clarify:

@Blackclaws :

Actually you have to make sure that you do not destroy the NativeMemory that you had to pass into that method before its actually finished...So if I want to reduce the memory usage of my application I have to make sure to wait for job completion in order to then clean up the NativeArray I had to pass in.

That's true, but you can schedule a job that deallocates a NativeArray using the JobHandle returned by ScheduleAddImageJob as its dependency. See DeallocateOnJobCompletionAttribute

It would be much easier if the API surface just took any of the image formats that Texture2D can load and handled all the Allocation/Deallocation behind the scenes.

We have exactly that :) See the extension method ScheduleAddImageJob(MutableRuntimeReferenceImageLibrary, Texture2D, String, Nullable<Single>, JobHandle)

@eugeneloza :

if (newLibrary is MutableRuntimeReferenceImageLibrary mutableLibrary) { jobHandle = MutableRuntimeReferenceImageLibraryExtensions.ScheduleAddImageJob(mutableLibrary, imageTexture, imageName, 1); }

This is a very cumbersome way to invoke that method. The point of extension methods is that they, well, extend an existing type. You can just do

jobHandle = mutableLibrary.ScheduleAddImageJob(imageTexture, imageName, 1);

[chaoslife]

Hello, our program will recognize the picture after entering AR, then exit AR to destroy the AR module. When entering AR for the first time, call the mutableLibrary.ScheduleAddImageJob method to correctly pass in the name of the image, and then in the trackedImagesChanged callback, the name and the GUID have values. After the AR is destroyed, enter again, the name returned by ARTrackedImage is empty, and the GUID is 00000?

Is this what is the reason

[Hless]

@chaoslife I have been struggling with exactly the same issue. See my earlier comment to resolve it: https://github.com/Unity-Technologies/arfoundation-samples/issues/385#issuecomment-650094655

[chaoslife]

@chaoslife我一直在努力解决同样的问题。请参阅我先前的评论以解决此问题： ＃385（评论）

Ok,Thanks.I'll try it

[Unity3D-Hardik]

Any solution for Bug mention in same Thread ? I am facing same issue

Our scenario is to be able to track image A, then image A+B+C, then remove image B from tracking. It is currently not possible. We also tried to track image A, destroy the ARTrackedImageManager, re-create it, re-configure its mutable image library, and adding the image to the mutable image library once again. What we end up with is only partially working: we properly receive the trackedImagesChanged callback, but the name of the ARImage and its texture are gone (probably badly cleaned native objects when destroying theARTrackedImageManager), even though the event itself returns accurate positions and rotations. We are currently investigating thoroughly this issue with a sample project and will submit a ticket if confirmed.

[marcochr]

An alternative to eugeneloza's solution is to to call LoaderUtility.Deinitialize() when the scene is unloaded. This is how Unity handles this in their image tracking sample in AR Foundation Samples.

See the SceneUtility.cs script.

using UnityEngine.SceneManagement;

namespace UnityEngine.XR.ARFoundation.Samples
{
    public class SceneUtility : MonoBehaviour
    {
        void Awake()
        {
            DontDestroyOnLoad(gameObject);
        }

        void OnEnable()
        {
            SceneManager.sceneUnloaded += OnSceneUnloaded;
        }

        void OnSceneUnloaded(Scene current)
        {
            if (current == SceneManager.GetActiveScene())
            {
                LoaderUtility.Deinitialize();
                LoaderUtility.Initialize();
            }
        }

        void OnDisable()
        {
            SceneManager.sceneUnloaded -= OnSceneUnloaded;
        }
    }
}

[Unity3D-Hardik]

I tried LoaderUtility but that won't work in my case !! I am not sure what is the reason. so after research, I got LoaderUtility to use the same class that I use so I went with that.