Output Prefab Name along with labels

[umairkhawaja]

I checked the custom labeler script in FAQs but that was tracking a single object in scene. Is it somehow possible to output prefab name along with class name in captures JSON?

[RuiyuZ]

Hi @umairkhawaja, Sorry I am not sure I fully understand your question, I will try to help as much as I can.

I am assuming you are looking at this script: https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/DatasetCapture.md

If you want to write out metadata for multiple objects, you can create a list of game objects like: public GameObject[] gameObjects; and place all the game objects you want to label into the list in the unity editor.

And then deal with these objects in the report metrics function, and use DatasetCapture.ReportMetric() to report the information you want. Here is some pseudocode to do so:

MetricDefinition myMetricDef = new MetricDefinition("blablabla", "blablabla");
void Start(){
    DatasetCapture.instance.RegisterMetric(myMetricDef);
}
void Update(){
       // Only report metrics during capture frames
        if (!perceptionCamera.SensorHandle.ShouldCaptureThisFrame)
        {
            return;
        }
     ReportObjectMetrics();
}
void ReportObjectMetrics(){
        // Gather metrics
        var prefabName = null;
        int className = null;
        foreach (var gameObject in gameObjects)
        {
            if (gameObject.activeInHierarchy)
            {
                // Get prefab name
                var prefabGameObject = PrefabUtility.GetCorrespondingObjectFromSource(gameObject)
                prefabName = prefabGameObject.name;
                // Get class name
                gameObject.GetType().Name
                break;
            }
        }
       myMetric myMetric = new myMetric
        {
            myPrefabName = prefabName 
            myClassName = className
        };
        string data = JsonUtility.ToJson(myMetric);
        // Write metrics
        DatasetCapture.instance.ReportMetric(myMetricDef, new GenericMetric(data, myMetricDef));
}

Please let me know if this helps. Otherwise, we can always get another pair of eyes on this. Thanks!

Best, Ruiyu

[umairkhawaja]

Apologies for the ambiguity, but these seems to be it. Will test it out

Thank you!

[umairkhawaja]

@RuiyuZ Hi, I just finished incorporating your pseudo-code. However, I found it easier to expand the TargetPositionDefinition annotation in https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/DatasetCapture.md

The issue that I'm facing is that it seems like the captures file only write one annotation per frame. Here's my piece of code, can you find any obvious mistake I'm doing?

`protected override void Setup() { lightMetricDefinition = new MetricDefinition( "LightMetric", "lightMetric1", "The world-space position of the light"); DatasetCapture.RegisterMetric(lightMetricDefinition);

    vehicleInformationDef = new VehicleInformationDef("vehicle_information");
    DatasetCapture.RegisterAnnotationDefinition(vehicleInformationDef);
}

protected override void OnBeginRendering(ScriptableRenderContext scriptableRenderContext)
{
    //Report the light's position by manually creating the json array string.
    var lightPos = targetLight.transform.position;
    var metric = new GenericMetric(new[] { lightPos.x, lightPos.y, lightPos.z }, lightMetricDefinition);
    DatasetCapture.ReportMetric(lightMetricDefinition, metric);

    foreach (var gameObject in sceneVehicles)
    {
        if (gameObject.activeInHierarchy)
        {
            // Get prefab name
            // var prefabGameObject = UnityEditor.PrefabUtility.GetCorrespondingObjectFromSource(gameObject);
            // var prefabName = prefabGameObject.name;
            var prefabName = gameObject.name;

            //compute the location of the object in the camera's local space
            Vector3 vehiclePos = perceptionCamera.transform.worldToLocalMatrix * gameObject.transform.position;

            // TODO: FIX THIS
            float vehicleRot = Quaternion.Angle(gameObject.transform.rotation, perceptionCamera.transform.rotation);

            //Report using the PerceptionCamera's SensorHandle if scheduled this frame
            var sensorHandle = perceptionCamera.SensorHandle;

            if (sensorHandle.ShouldCaptureThisFrame)
            {
                var annotation = new VehicleInformation(vehicleInformationDef, sensorHandle.Id, vehiclePos, vehicleRot, prefabName);
                sensorHandle.ReportAnnotation(vehicleInformationDef, annotation);
            }
        }
    }

}`

[umairkhawaja]

I also tried adding it as a Metric Definition, but even outputs a single name and moreover, there's no correspondence with the instance_id in the metrics.json file

[mkamalza]

Hi,

That is the expected behavior. Only one annotation json block can be captured per annotation definition in each captured frame. To capture information from multiple objects, you will need to combine the information into one object and write it once using a single call to sensorHandle.ReportAnnotation in the OnBeginRendering function. You can use a single List<vehicleInformationDef> to store information from all vehicles in each frame.

In order to map these objects to their instance ids, you can fetch the Labeling component of the object, and if it is active, read the instanceId field of it:

var labeling = gameObject.GetComponent<Labeling>();
if (labeling.enabled)
{
    vehicleInformationDef.instanceId = labeling.instanceId;
}

[umairkhawaja]

Hi @mkamalza , thank you for clarifying that. However, I'm still unsure how to correct my problem. Some confusions

1) What's the difference between Annotation and AnnotationDefinition? For each instance of a game object, should I create a new Annotation or AnnotationDefinition?

2) To store information about all vehicles in each frame, should I use a List of Annotations or AnnotationDefinitions?

3) ReportAnnotation expects an Annotation Definition and an Annotation, so how do I report the List containing the relevant information?

Thanks!

[mkamalza]

Hi @umairkhawaja

An Annotation is one block of data that is recorded whenever the ReportAnnotation method is called, whereas an AnnotationDefinition defines the stream of data that the annotation belongs to. In a sense, it is the "type" of the annotations recorded.

In this case, your annotation would need to include information on all the vehicles. I created a sample labeler for you that records information about all labeled objects in the scene. Copy paste this into a file named LabeledObjectInfoLabeler.cs and then you can add the labeler to your Perception Camera. You can modify this code slightly to filter out specific objects if you don't want all labeled objects, or write additional information besides object names and instanceIds.

using System;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.Perception.GroundTruth;
using UnityEngine.Perception.GroundTruth.DataModel;
using UnityEngine.Rendering;
using Object = UnityEngine.Object;

[Serializable]
public class LabeledObjectInfoLabeler : CameraLabeler
{
    AnnotationDefinition m_AnnotationDefinition;
    public string annotationId = "94179c03-6258-4cfe-8449-f337fcd24301";
    public override string description => "Information on all labeled objects in the scene that have active labeling components.";
    public override string labelerId => "LabeledObjectInfo";
    protected override bool supportsVisualization => false;

    class LabeledObjectInfoAnnotationDefinition : AnnotationDefinition
    {
        public LabeledObjectInfoAnnotationDefinition(string id)
            : base(id) { }

        public override string modelType => "LabeledObjectInfoAnnotationDefinition";
        public override string description => "Information on all labeled objects in the scene that have active labeling components.";
    }

    [Serializable]
    class LabeledObjectInfoAnnotation : Annotation
    {
        public LabeledObjectInfoAnnotation(AnnotationDefinition definition, string sensorId, List<ObjectInfo> labeledObjectsInScene)
            : base(definition, sensorId)
        {
            this.labeledObjectsInScene = labeledObjectsInScene;
        }

        public List<ObjectInfo> labeledObjectsInScene;
        public override bool IsValid() => true;
    }

    public class ObjectInfo
    {
        public string objectName;
        public uint instanceId;
    }

    protected override void Setup()
    {
        m_AnnotationDefinition = new LabeledObjectInfoAnnotationDefinition(annotationId);
        DatasetCapture.RegisterAnnotationDefinition(m_AnnotationDefinition);
    }

    protected override void OnBeginRendering(ScriptableRenderContext scriptableRenderContext)
    {
        if (sensorHandle.ShouldCaptureThisFrame)
        {
            var infoList = new List<ObjectInfo>();
            var labelings = Object.FindObjectsOfType<Labeling>();

            foreach (var labeling in labelings)
            {
                if (labeling.enabled && labeling.gameObject.activeSelf)
                {
                    var gameObject = labeling.gameObject;

                    infoList.Add(new ObjectInfo
                    {
                        instanceId = labeling.instanceId,
                        objectName = gameObject.name
                    });
                }
            }

            var annotation = new LabeledObjectInfoAnnotation(m_AnnotationDefinition, sensorHandle.Id, infoList);
            sensorHandle.ReportAnnotation(m_AnnotationDefinition, annotation);
        }
    }
}

Here is a sample json block output from this code when used in the Perception HDRP test project:

{
          "labeled_objects_in_scene": [
            {
              "object_name": "Crate",
              "instance_id": 1
            },
            {
              "object_name": "Terrain",
              "instance_id": 2
            },
            {
              "object_name": "Cube",
              "instance_id": 3
            },
            {
              "object_name": "Box",
              "instance_id": 4
            }
          ],
          "annotation_id": "94179c03-6258-4cfe-8449-f337fcd24301",
          "model_type": "LabeledObjectInfoAnnotationDefinition",
          "description": "Information on all labeled objects in the scene that have active labeling components.",
          "sensor_id": "camera",
          "id": "94179c03-6258-4cfe-8449-f337fcd24301"
        },
        {
          "id": "12f94d8d-5425-4deb-9b21-5e53ad957d66",
          "annotation_definition": "12f94d8d-5425-4deb-9b21-5e53ad957d66",
          "filename": "SemanticSegmentation59cca4d0-7965-40c6-8e2e-8e1608f7d59d/segmentation_2.png"
        }
}

[umairkhawaja]

@mkamalza This is exactly it! Thank you very much for your efforts. I could easily extend this snippet to add other information as well. Closing the issue now