Semantic Segmentation Labels don't show up randomly at some Iterations

[ltacke]

Hi there,

I'm currently working with the Perception Package to generate Data of a Concrete Wall with different types of Cracks in front of it. I want these pictures to be labeled by the SemanticSegmentation Labeler.

I got nearly everything to work, but now I' facing some strange issues with the Simulation Run, as you can see in the video:

https://user-images.githubusercontent.com/90914166/232129631-2038c334-53fb-417a-a789-99e83025bdc9.mov

As you could see, at some iterations it get's the annotations of the crack and sometimes randomly not. I tried to find a pattern but couldn't find one. It doesn't depend on the crack that is used or of the number of iterations etc.

To give the best overview of what I have done, I would now share all the relevant components and code snippets so I will hopefully find somebody that could help me solve that problem 😄

Main Camera:

Crack-Prefab: (They are all the same - I triple checked 😉)

The Scenario Definition:

The Self Written Randomizer for loading the Crack-Prefab:

using System;
using UnityEngine;
using UnityEngine.Perception.Randomization.Parameters;
using UnityEngine.Perception.Randomization.Randomizers;
using UnityEngine.Perception.Randomization.Samplers;

// Add this Component to any GameObject that you would like to be randomized. This class must have an identical name to
// the .cs file it is defined in.
public class NewRandomizerTag : RandomizerTag {}

[Serializable]
[AddRandomizerMenu("Crack Randomizer")]
public class CrackRandomizer : Randomizer
{
    private GameObject currentInstance;

    public CategoricalParameter<GameObject> prefabs;
    public Vector3Parameter crackRotation;

    protected override void OnIterationStart()
    {
            currentInstance = GameObject.Instantiate(prefabs.Sample());
            //currentInstance.transform.rotation = Quaternion.Euler(crackRotation.Sample());
    }

    protected override void OnIterationEnd()
    {
        GameObject.Destroy(currentInstance);
    }

}

Editor Version: 2021.3.17f1

You really builded an awesome Package for Unity, which I enjoy to use so far!

I really hope there is somebody who could help me to solve my problem!

[ltacke]

Update:

I tried it with the newest LTS Version of the Unity Editor: 2021.3.23f1 and I set everything up from scratch.

Still the same issue: Sometimes the labels, still randomly, show up and sometimes they don't.

I would really appreciate if somebody of the authors would have a look at that 😃

https://user-images.githubusercontent.com/90914166/232779670-d84bd5a3-43ed-4030-a120-fd1a312e6abf.mov

[ltacke]

Hey dear Contributors and Community, I still struggle to find any solution or hint why that error occurs. To find a solution has a really big impact on my study I'm doing right now - so it would be really awesome to have somebody have a look at this.

@mkamalza @sleal-unity @JonathanHUnity Sorry for the noise guys - I don't know how to help myself beyond that point I've come.

[ltacke]

I played around a bit more and tested the Crack Prefabs by themselves - I now think that the CustomRandomizer is the one causing that problem.

using System;
using UnityEngine;
using UnityEngine.Perception.Randomization.Parameters;
using UnityEngine.Perception.Randomization.Randomizers;
using UnityEngine.Perception.Randomization.Samplers;

// Add this Component to any GameObject that you would like to be randomized. This class must have an identical name to
// the .cs file it is defined in.
public class NewRandomizerTag : RandomizerTag {}

[Serializable]
[AddRandomizerMenu("Crack Randomizer")]
public class CrackRandomizer : Randomizer
{
    private GameObject currentInstance;

    public CategoricalParameter<GameObject> prefabs;
    public Vector3Parameter crackRotation;

    protected override void OnIterationStart()
    {
            currentInstance = GameObject.Instantiate(prefabs.Sample());
            //currentInstance.transform.rotation = Quaternion.Euler(crackRotation.Sample());
    }

    protected override void OnIterationEnd()
    {
        GameObject.Destroy(currentInstance);
    }

}

But still haven't figured out why that is the case. Still looking out for some help :)

[ltacke]

After looking at some other (slightly comparable) Issues here, I found that I oversaw the code for the Randomizer Tag in the first place. So I decided to split them up into to scripts (just how the default ones are defined). So now there is: CrackRandomizer.cs

using System;
using UnityEngine;
using UnityEngine.Perception.Randomization.Parameters;
using UnityEngine.Perception.Randomization.Randomizers;
using UnityEngine.Perception.Randomization.Samplers;

// Add this Component to any GameObject that you would like to be randomized. This class must have an identical name to
// the .cs file it is defined in.
//[AddComponentMenu("Perception/RandomizerTags/Crack Randomizer Tag")]
//public class CrackRandomizerTag : RandomizerTag {}

[Serializable]
[AddRandomizerMenu("Crack Randomizer")]
public class CrackRandomizer : Randomizer
{
    private GameObject currentInstance;

    public CategoricalParameter<GameObject> prefabs;
    public Vector3Parameter crackRotation;

    protected override void OnIterationStart()
    {
            currentInstance = GameObject.Instantiate(prefabs.Sample());
            //currentInstance.transform.rotation = Quaternion.Euler(crackRotation.Sample());
    }

    protected override void OnIterationEnd()
    {
        GameObject.Destroy(currentInstance);
    }

}

And the CrackRandomizerTag.cs

namespace UnityEngine.Perception.Randomization.Randomizers.Tags
{
   [AddComponentMenu("Perception/RandomizerTags/Crack Randomizer Tag")]
    public class CrackRandomizerTag : RandomizerTag {}
}

Now I was able to add that RandomizerTag to all of my prefabs. (Side note: I really hoped that changed everything and the error is gone) Buuut, it still does the same weird thing.

Just adding a video of a frame by frame simulation and showcasing that every prefab has the CrackRandomizerTag attached to it. Still the error occurs completely random

https://user-images.githubusercontent.com/90914166/234645521-f22e2f15-1438-4276-9d7f-5197252d58d7.mov