Problems when adding a labeler to a perception camera

[wwwwwyyyyyxxxxx]

Hi! I'm using the Perception package in a URP project, so I'm using the 0.11 version.

I want to label objects spawned by users at runtime and refer to the suggestions in #467. When there are no objects spawned dynamically, I can create a perception camera and add a labeler to it. However, after destroying the old perception camera and creating a new perception camera in order to label newly generated objects, I fail to attach a labeler to it. My code looks like

var followCameraObject = GameObject.FindGameObjectWithTag("camBase");
if (followCameraObject.GetComponent<PerceptionCamera>() != null) {
    Destroy(followCameraObject.GetComponent<PerceptionCamera>());
}
followCameraObject.AddComponent<PerceptionCamera>();
followCameraObject.GetComponent<PerceptionCamera>().AddLabeler(semanticSegmentationLabeler); 

I also tried not to destroy the perception camera but to remove the labeler, but I got NullReferenceException when I use code like

var followCameraObject = GameObject.FindGameObjectWithTag("camBase");
followCameraObject.GetComponent<PerceptionCamera>().RemoveLabeler(followCameraObject.GetComponent<PerceptionCamera>().m_lablers[0]);

Would you like to give me some advice on how to label the runtime-spawned objects via script?

Appreciate your help!

[mark1ng123]

Hey it is really hard to follow your code, can you edit the issue and fix the code identation please. I did attach daynamiclly the lablers the steps are:

1. Get the componenet for the perception camera, Camera=GameObject.Find("MainCamera").GetComponent<PerceptionCamera>();
2. Create a labling config object, var Config = ScriptableObject.CreateInstance<UnityEngine.Perception.GroundTruth.LabelManagement.IdLabelConfig>();
3. Add a labeling counter if you have many labels, int labelCounter = 1
4. Then attach them to a label management list,

   List<UnityEngine.Perception.GroundTruth.LabelManagement.IdLabelEntry> labels; 
   foreach (string label in labels)
       {
           labels.Add(new UnityEngine.Perception.GroundTruth.LabelManagement.IdLabelEntry{id = labelCounter, label = label});
           labelCounter++;
       }

5. intialize the config with the list you created: Config.Init(entries);
6. Then add the corect labler in your case is semantic segmentation so : Camera.AddLabeler(new UnityEngine.Perception.GroundTruth.Labelers.SemanticSegmentationLabeler(Config)); And thats it you have a labler set and ready to run. Note: dont forget to add the right lables to the gameobjects, you can also do it during runtime, i also recommend it to give a more dynamic feel

[wwwwwyyyyyxxxxx]

Hi mark1ng123!

Thanks for your reply! Sorry for the code indentation problem, I have already fixed it.

And for dynamically adding labels, my dynamically addling labels code looks exactly like your code. My main problem actually is that my project spawns objects very often, so I have to make a new labeler containing the newly generated object and add it to the perception camera simultaneously. This could cause the binary running extremely slow.

Therefore, I have to

1. destroy the perception camera and recreate one

   var followCameraObject = GameObject.FindGameObjectWithTag("camBase");
   if (followCameraObject.GetComponent<PerceptionCamera>() != null) {
   Destroy(followCameraObject.GetComponent<PerceptionCamera>());
   }
   followCameraObject.AddComponent<PerceptionCamera>();
   followCameraObject.GetComponent<PerceptionCamera>().AddLabeler(semanticSegmentationLabeler); 

   or

2. remove the current labeler and readd one

   var followCameraObject = GameObject.FindGameObjectWithTag("camBase");
   followCameraObject.GetComponent<PerceptionCamera>().RemoveLabeler(followCameraObject.GetComponent<PerceptionCamera>().m_lablers[0]);

   as mentioned in #467.

However, I got stuck with

1. cannot adding labeler to the recreated perception camera as I found the newly created camera did not have any labeler in the editor.
2. failing to remove the current labeler with a NullReferenceException. But I have checked

   followCameraObject.GetComponent<PerceptionCamera>()
   followCameraObject.GetComponent<PerceptionCamera>().m_lablers[0]

   are not null. respectively.

Appreciate your help!

Hey it is really hard to follow your code, can you edit the issue and fix the code identation please. I did attach daynamiclly the lablers the steps are:

1. Get the componenet for the perception camera, Camera=GameObject.Find("MainCamera").GetComponent<PerceptionCamera>();
2. Create a labling config object, var Config = ScriptableObject.CreateInstance<UnityEngine.Perception.GroundTruth.LabelManagement.IdLabelConfig>();
3. Add a labeling counter if you have many labels, int labelCounter = 1
4. Then attach them to a label management list,

List<UnityEngine.Perception.GroundTruth.LabelManagement.IdLabelEntry> labels; 
foreach (string label in labels)
        {
            labels.Add(new UnityEngine.Perception.GroundTruth.LabelManagement.IdLabelEntry{id = labelCounter, label = label});
            labelCounter++;
        }

5. intialize the config with the list you created: Config.Init(entries);
6. Then add the corect labler in your case is semantic segmentation so : Camera.AddLabeler(new UnityEngine.Perception.GroundTruth.Labelers.SemanticSegmentationLabeler(Config)); And thats it you have a labler set and ready to run. Note: dont forget to add the right lables to the gameobjects, you can also do it during runtime, i also recommend it to give a more dynamic feel

[mark1ng123]

@wwwwwyyyyyxxxxx Dont destroy your perception camera, just set a new label config, when youll set a new label config with the newly genrated objects it will clean it up. Another approach is to set all the labels that will spawn onAwake(), and then youll add all the labels of your objects and you will not need to touch this component. Write a function that reads all the labels youll need during the simulation and then add all of them to the label config. If you choose the first approach everytime you will set a new label config it will clean up the last one that was used it is in the setup function it the labler perception class.

[wwwwwyyyyyxxxxx]

@mark1ng123 Thanks for your advice!

For the first approach, I only found API AddLabeler in https://github.com/Unity-Technologies/com.unity.perception/blob/760dc0e597dae0b5349e12b74efbf141cb8cdee2/com.unity.perception/Runtime/GroundTruth/PerceptionCamera.cs#L201 but haven't found something like SetLabeler?

Well for sure, added a new label config, the camera will re-render the scene with new config. But the configs are still in the perception camera's m_labelers. After adding a lot of label configs (like 10 or 20), it causes huge slowdown since the perception camera needs to view all its labelers to decide the segmentation color.

For the second approach, unfortunately, my project allows users generate as many arbitrary objects as they want and segments each object in a different color. So I didn't find a way to prepare a config containing all the labels I need.

Appreciate your help!

@wwwwwyyyyyxxxxx Dont destroy your perception camera, just set a new label config, when youll set a new label config with the newly genrated objects it will clean it up. Another approach is to set all the labels that will spawn onAwake(), and then youll add all the labels of your objects and you will not need to touch this component. Write a function that reads all the labels youll need during the simulation and then add all of them to the label config. If you choose the first approach everytime you will set a new label config it will clean up the last one that was used it is in the setup function it the labler perception class.

[mark1ng123]

@wwwwwyyyyyxxxxx Then i would recommend working with the RemoveLabeler method in the perception camera game object, everytime you add the labelers save them in the list, meaning every iteration save them in a list and every end of iteration iterate over them like so:

foreach(var labler in lablers){
   camera.RemoveLabeler(labler);
}

So every start of iteration you will add the lablers to a list so youll have a pointer reference of those and every end of iteration you will remove using the pointer refernce assitance. Another way to go: if you look closely in the lablers object there is a way to set a label list in the intialization cfg.Init(new List<IdLabelEntry> {new IdLabelEntry {id = 1, label = "test"}}); So instead of giving a hard coded list youll set a list of idlabel entries and change the config of exsiting labler every iteration, and it will update in the component of the perception camera .

[wwwwwyyyyyxxxxx]

Hi @mark1ng123 ! Thanks for your suggestions!

For the first approach, actually I have tried RemoveLabeler API in the origin issue. Sorry for the indentation problem at first.

My code looks like

var followCameraObject = GameObject.FindGameObjectWithTag("camBase");
followCameraObject.GetComponent<PerceptionCamera>().RemoveLabeler(followCameraObject.GetComponent<PerceptionCamera>().m_lablers[0]);

but I got NullReferenceException on the second line while I have checked all related objects are not null pointer. Would you like to give me some suggestions on that?

For the second approach, refer to #467 , @JonathanHUnity said that "changing a LabelConfig at runtime is only possible before the config has been used the first time." I'm not sure if you want to change the LabelConfig of the perception camera during runtime?

Appreciate your help!

@wwwwwyyyyyxxxxx Then i would recommend working with the RemoveLabeler method in the perception camera game object, everytime you add the labelers save them in the list, meaning every iteration save them in a list and every end of iteration iterate over them like so:

foreach(var labler in lablers){
   camera.RemoveLabeler(labler);
}

So every start of iteration you will add the lablers to a list so youll have a pointer reference of those and every end of iteration you will remove using the pointer refernce assitance. Another way to go: if you look closely in the lablers object there is a way to set a label list in the intialization cfg.Init(new List<IdLabelEntry> {new IdLabelEntry {id = 1, label = "test"}}); So instead of giving a hard coded list youll set a list of idlabel entries and change the config of exsiting labler every iteration, and it will update in the component of the perception camera .

[mark1ng123]

@wwwwwyyyyyxxxxx I did suggest you another approach, dont use m_lablers at index 0 to remove, thats why you get null reference, save a reference to the config, save it in a variable. Read carefully what i said, when adding a new labeler to the camera devide it to two steps:

1. Save the labler in a variable. (I suggested a list of lablers)
2. Set it on the perception camera using add labeler the variable. Then you will have a reference to the labler and you could remove it.

Look at the bold part:

@wwwwwyyyyyxxxxx Then i would recommend working with the RemoveLabeler method in the perception camera game object, everytime you add the labelers save them in the list, meaning every iteration save them in a list and every end of iteration iterate over them like so:

foreach(var labler in lablers){
   camera.RemoveLabeler(labler);
}

So every start of iteration you will add the lablers to a list so youll have a pointer reference of those and every end of iteration you will remove using the pointer refernce assitance. Another way to go: if you look closely in the lablers object there is a way to set a label list in the intialization cfg.Init(new List<IdLabelEntry> {new IdLabelEntry {id = 1, label = "test"}}); So instead of giving a hard coded list youll set a list of idlabel entries and change the config of exsiting labler every iteration, and it will update in the component of the perception camera .

[wwwwwyyyyyxxxxx]

Thanks for your reply!

Actually, I have tried the way to save it in a variable in the way like saving labeler:

var followCameraObject = GameObject.FindGameObjectWithTag("camBase");
if (followCameraObject.GetComponent<PerceptionCamera>() == null)
{
    followCameraObject.AddComponent<PerceptionCamera>();
}
var perceptionCamera = followCameraObject.GetComponent<PerceptionCamera>();
perceptionCamera.simulationDeltaTime = 0.2f;
perceptionCamera.AddLabeler(semanticSegmentationLabeler);

_builder.currentCameraLabeler = semanticSegmentationLabeler;

And remove labeler like

var perceptionCamera = followCameraObject.GetComponent<PerceptionCamera>();
perceptionCamera.simulationDeltaTime = 0.2f;
perceptionCamera.AddLabeler(semanticSegmentationLabeler);
if (AB.currentCameraLabeler != null)
{
    perceptionCamera.RemoveLabeler(AB.currentCameraLabeler);
}
AB.currentCameraLabeler = semanticSegmentationLabeler;

where AB is _builder.

But I still get NullReferenceException.

I'm sure it comes from perceptionCamera.RemoveLabeler(AB.currentCameraLabeler), since NullReferenceException goes away once I comment this line.

Appreciate your help!

@wwwwwyyyyyxxxxx I did suggest you another approach, dont use m_lablers at index 0 to remove, thats why you get null reference, save a reference to the config, save it in a variable. Read carefully what i said, when adding a new labeler to the camera devide it to two steps:

1. Save the labler in a variable. (I suggested a list of lablers)
2. Set it on the perception camera using add labeler the variable. Then you will have a reference to the labler and you could remove it.

Look at the bold part:

@wwwwwyyyyyxxxxx Then i would recommend working with the RemoveLabeler method in the perception camera game object, everytime you add the labelers save them in the list, meaning every iteration save them in a list and every end of iteration iterate over them like so:

foreach(var labler in lablers){
   camera.RemoveLabeler(labler);
}

So every start of iteration you will add the lablers to a list so youll have a pointer reference of those and every end of iteration you will remove using the pointer refernce assitance. Another way to go: if you look closely in the lablers object there is a way to set a label list in the intialization cfg.Init(new List<IdLabelEntry> {new IdLabelEntry {id = 1, label = "test"}}); So instead of giving a hard coded list youll set a list of idlabel entries and change the config of exsiting labler every iteration, and it will update in the component of the perception camera .

[mark1ng123]

It looks like you are missing something out, can you show me how do you declare the labler ? It should work, I used it exactly like i described

[wwwwwyyyyyxxxxx]

Thanks for your reply!

I define the labeler like

var labelConfig = ScriptableObject.CreateInstance<SemanticSegmentationLabelConfig>();
var semanticSegmentationLabelEntries = new List<SemanticSegmentationLabelEntry>();
foreach (var item in _builder.gameObjectLabelCount)
{
    for (int i = 0; i < item.Value; i++)
    {
        semanticSegmentationLabelEntries.Add(
            new SemanticSegmentationLabelEntry()
            {
                label = item.Key + i.ToString(),
                color = new Color32((byte)Random.Range(0, 256), (byte)Random.Range(0, 256), (byte)Random.Range(0, 256), 255)
            }
        );
    }
}
labelConfig.Init(semanticSegmentationLabelEntries);
var semanticSegmentationLabeler = new SemanticSegmentationLabeler(labelConfig);

the _builder.gameObjectLabelCount is a dictionary of the count of objects with the same name, whose label is name+its serial number. And semanticSegmentationLabeler is the same in my previous code in my previous reply.

Appreciate your help!

It looks like you are missing something out, can you show me how do you declare the labler ? It should work, I used it exactly like i described

[mark1ng123]

Super weird it look fine, very similar to my approach, and when i used RemoveLabeler my code worked. Maybe the RemoveLabeler works and you try to approach the labler you already removed in some other place which throws a null reference exception. Can you check if you are calling the labler you removed in some other place or maybe the iteration of perception is trying to call it?

[wwwwwyyyyyxxxxx]

Thanks for your help!

I will check it. Actually Perception supports remove non-null labels that has been remove. It will return false but not NullReferenceException. Maybe you are right, the iteration of perception is trying to call a removing labeler.

I tried another way to implement it: I create a camera once new objects spawned, add PerceptionCamera and labelconfig to it and destroy the original camera. Although it may flick when changing camera.

Thanks for your enthusiastic help!

Super weird it look fine, very similar to my approach, and when i used RemoveLabeler my code worked. Maybe the RemoveLabeler works and you try to approach the labler you already removed in some other place which throws a null reference exception. Can you check if you are calling the labler you removed in some other place or maybe the iteration of perception is trying to call it?

[mark1ng123]

Glad to help @wwwwwyyyyyxxxxx hope youll get it right, I really hate destroying objects during an iteration you need to be super careful it always brings the null reference exception which i hate to debug (Really remind me the segmentation fault in C ).

[wwwwwyyyyyxxxxx]

@mark1ng123 You are absolutely right. It will be great that Perception supports dynamic config labeler.