How to spawn a prefab to match the tracked 3D object in the camera view?

[michelepanegrossi]

I am testing the 3D object tracking feature.

I have successfully generated the arobject and added it to the reference library. The tracking seems to work. When generating the arobject with the provided app, I had to select the object origin which I roughly placed to match the origin of the 3D scan I have of the same object.

The next step would be to spawn a prefab to overlay onto the recognised object, however I don't understand how to achieve this.

I have a 3D scan of the object obtained with photogrammetry. However, I am not sure what is the size of the arobject (or the ARTrackedObject) in relation to my 3D scan.

I was able to obtain the Transform of the ARTrackedObject (which I assume is the world position of the model recognised by ARKit), using this code:

using System.Collections.Generic;
using UnityEngine;
using UnityEngine.XR.ARFoundation;
using UnityEngine.UI;

[RequireComponent(typeof(ARTrackedObjectManager)) ]
public class ObjectTrackingInfo : MonoBehaviour
{
    [SerializeField]
    private GameObject[] _placeablePrefabs;

    private ARTrackedObjectManager _trackedObjectManager;

    public Text _debugtext;

    GameObject newPrefab;

    private void Awake()
    {
        _trackedObjectManager = FindObjectOfType<ARTrackedObjectManager>();
    }

    private void OnEnable()
    {
        _trackedObjectManager.trackedObjectsChanged += ObjectChnaged;
    }

    private void OnDisable()
    {
        _trackedObjectManager.trackedObjectsChanged -= ObjectChnaged;

    }

    private void ObjectChnaged(ARTrackedObjectsChangedEventArgs eventArgs)
    {
        foreach(ARTrackedObject trackedObject in eventArgs.added)
        {
            newPrefab = Instantiate(_placeablePrefabs[0], trackedObject.transform.position, trackedObject.transform.rotation);
        }

        foreach (ARTrackedObject trackedObject in eventArgs.updated)
        {
            UpdateModel(trackedObject);
        }

        foreach (ARTrackedObject trackedObject in eventArgs.removed)
        {
           // _spwnedprefabs[trackedObject.name].SetActive(false);
        }

    }

    private void UpdateModel(ARTrackedObject _trackedObject)
    {
        string name = _trackedObject.referenceObject.name;

        Vector3 pos = _trackedObject.transform.position;
        Quaternion rot = _trackedObject.transform.rotation;
        Vector3 scale = _trackedObject.transform.localScale;

        _debugtext.text = $" Name: {name}\n Pos: {pos}\n Rot: {rot}\n Scale: {scale}";

        newPrefab.transform.position = pos;
        newPrefab.transform.rotation = rot;
    }
}

but the spawned prefab doesn't match at all the object in the camera view.

If I understand correctly, the ARTrackedObjectManager spawns either the Tracked Object Prefab or an empty object but where is this spawned? At the origin of the arobject? And what is the hierarchy? Is the prefab spawned as a child of something else?

When I retrieve the transform of the ARTrackedObject, where is that GameObject in the hierarchy?

[michelepanegrossi]

@tdmowrer are you able to comment on this?

[michelepanegrossi]

If anybody is interested, this seems to be something to be solved in the Apple ARKit scanning app which is necessary to create arobject assets.

https://developer.apple.com/documentation/arkit/content_anchors/scanning_and_detecting_3d_objects

The important thing is to have a representation of the object you want to scan (3D model or 3D scan) BEFORE you start scanning the object to create the arobject. This model MUST be in USDZ format. I have created mine with Polycam but you can also use Apple Object Capture https://developer.apple.com/videos/play/wwdc2021/10076/

When you finish scanning the object to create the arobject you have the option to load a model. Load and select the USDZ model previously saved on your device.

At this point you need to adjust the position, rotation and scale of the 3D model MANUALLY (Seriously!?!?!?! Yes....) to match the real world object you see in the camera view.

Once you have done this though, the arobject will be automatically resized to match the Transform you have manually applied to the 3D model.

At this point save the arobject.

When you import it in unity, you can simply use the ARObjectManager component to spawn the 3D model of the object (if you have only one object in your reference library).

I had to rotate mine 180 degrees on the Y though.

It sort of works, but the fact that the transform of the 3D model has to be adjusted manually really reduces the precision of the final result, especially if you have a 3D scan and not a CAD model of the object you are trying to detect.

I hope this helps. If anybody has more info or a better workflow I would love to hear about it!

Michele

[DavidMohrhardt]

Thanks for the write up!