how to eliminate 'ghost' labels? (detection objects that are not really visible but receive a label)

[lessw2020]

I have noticed that I am getting some degree of 'ghost labels' and wondering how to avoid that?

1 - I refer to a 'ghost label' as when I get a label showing up on the image, likely due to an object being buried within the distraction layer close to the surface but not actually visible (or at times, just barely visible). Regardless of reason, it's labelling that will detract from the AI training as there is not enough signal there for it to learn from.

2 - This issue may be caused by my need to wrap my detection objects inside a game object in order to have a local and global rotation, rather than simply the object itself...I thus have all of my desired objects laying flat in the scene at the start, and then are moved in and out based on the foreground randomizer.
Thus, is the fix that I need to do everything in code for this instead of having them already in the scene? Or is there a need to reset the state image to image? Most images don't have it but it's enough that it's a concern.

I'm attaching an image showing what I mean. You can see that while most labels are valid, there are a few (circled in red) where there is no object readily apparent but it has a label. In some cases I can see a tiny sliver of the object peeking through the distraction layer, so it's not a totally random label but for the obj detection, there is not enough visible for it to be a legit signal to learn from and thus a label is not a valid option. (see the bottom most red circle...there is an object peeking out of the distraction layer, but not enough to be valid).

Anyway, any input on how to avoid/fix this would be appreciated to ensure that I'm training only with valid labels.

[sleal-unity]

Hi again Less!

I'm not sure I completely understand the issue, but maybe a few follow questions will help me piece things together:

1. Can you walk me through the "wrap my detection objects inside a game object" part of your explanation? Maybe a screenshot of the relationship between these GameObjects in your scene's hierarchy and some additional context will help me understand your need for this intermediate wrapper object.

2. For your object placement strategy, did you modify the existing ForegroundObjectPlacementRandomizer or create your own Randomizer to vary the distance of placed foreground objects from the camera? Or are you randomizing the scale of already placed objects? And if the former is the case, are you indicating that some of these objects are being placed behind the background layer of distractor objects and that on occasion they peak through and get labeled when they shouldn't?

3. On a separate note, are you looking to allow different foreground objects to occlude over each other, or do you want them all to never overlap?

[lessw2020]

Hi @sleal-unity! Thanks for the fast reply. Here's some updates to help clarify:

1 - wrapping detection objects - the background on this is I have to ensure the rdts (objects I want to detect) always face forward as we don't/can't run detection on these based on side view or back. As a result, per another issue here, I was hitting gimbal lock trying to force them to face forward but then rotating them 360 while facing forward. The fix was to wrapper each item in a game object and setup global rotation on the game object (X = -90 makes them face to the camera...technically I'm doing -94 to -87 to add a bit of wobble). then I rotate on the Y axis locally to impart random clock face type spin to the object. As a result though, I have these placed in the scene and then add the gameobject to the foreground placement. Thus, this is different simply dropping prefabs in the foreground randomizer and may be part of this issue?

2 - I am using the default foreground randomizer and am using the default -3 distance. I am randomizing the scale though I placed this and rotation randomizers above the foreground placement with the idea of ensuring that the object is both scaled and rotated before being placed.

3 - Prefer not to have any occlusion at the foreground layer. I am using GridMax seperately as an augmentation to provide some occlusion while ensuring enough info that it can realistically id.

Attached is a screenshot of my setup - hopefully that helps clarify things a bit?

[sleal-unity]

So one thing I can recommend is to make your top level GameObjects (ex: go_rdt_nhs_strong) prefabs instead of just a GameObjects in your scene. Prefabs are like GameObject templates and do not exist in the scene until they are instantiated. The ForegroundObjectPlacmentRandomizer is coded such that it will use prefabs to place and destroy instances of objects it uses every frame and is not meant to manipulate GameObjects that are already present in the scene. In your ForegroundObjectPlacmentRandomizer though, you have references to raw GameObjects instead of prefabs (see the grey vs. blue icons, grey being a raw GameObject and blue being a prefab).

To do this, you simply need to drag all detection GameObjects (like go_rdt_nhs_strong) from the hierarchy window down into your project window. Once you've done this, the go_rdt_nhs_strong should be blue in your hierarchy to show that it is now an instance of a prefab somewhere in your project window. Next, delete all detection object prefab instances from your scene, then add only references to these prefabs to your ForegroundObjectPlacmentRandomizer.

The randomizer's prefabs list should look like this:

1. go_rdt_nhs_strong
2. go_rdt_nhs_invalid
3. go_accessbio_strong
4. ....

Also, after doing this, make sure to move your ForegroundObjectPlacementRandomizer back up the randomizer stack in your scenario. It needs to spawn the RDTs before the other randomizations will be effective.

[sleal-unity]

On a separate note, hit me up if you need an example of how to code a randomizer that places and scales objects without allowing for them to overlap (it's nothing too sophisticated, just a combo of two existing randomizers), but I think simply tuning up your separation distance on your foreground placement randomizer should do the trick for now.

[lessw2020]

Hi @sleal-unity - thanks for the info. Converting the gameobjects into prefabs appears to have resolved it! (I'll make a large dataset this afternoon to double check, but sample runs right now no longer show any so looks good. And good to know we should not have anything existing in the scene).
Also thanks for the tip regarding ForegroundObjectPlacementRandomizer coming first in the process stack..I was thinking the opposite, that the objects needed to be rotated and scaled behind the scene first in order to determine placement, but have adjusted that order in my setup to reflect Foreground should be first. Would definitely be really helpful to get an example of coding a randomizer to avoid overlap if possible - thanks very much for that.
I'm tuning the spacing distance now, but it does seem like having it in code would be nicer/more accurate, rather than hand tuning it for each new rdt.

[sleal-unity]

Okay, so this example randomizer will definitely ensure that no objects overlap, but what it will not do is pack objects close together. I'll put an item on my backlog for a creating an example placement randomizer that will pack the screen more thoroughly. Regardless, what I'm doing in this randomizer is:

1. Looping through all the assigned prefabs
2. Calculating their maximum bounds radius
3. Augmenting this radius with the maximum possible scale factor
4. Using this max radius as the separation distance in the Poisson Disk placement algorithm to ensure that no two objects overlap

using System;
using System.Linq;
using UnityEngine;
using UnityEngine.Perception.Randomization.Parameters;
using UnityEngine.Perception.Randomization.Randomizers;
using UnityEngine.Perception.Randomization.Randomizers.Utilities;
using UnityEngine.Perception.Randomization.Samplers;

[Serializable]
[AddRandomizerMenu("Example/No Overlap Foreground Object Placement Randomizer")]
public class NoOverlapForegroundObjectPlacementRandomizer : Randomizer
{
    /// <summary>
    /// The Z offset component applied to the generated layer of GameObjects
    /// </summary>
    public float depth;

    /// <summary>
    /// The maximum possible scale factor that may be applied to the placed objects in other downstream randomizers
    /// </summary>
    public float maxScaleFactor = 1f;

    /// <summary>
    /// The size of the 2D area designated for object placement
    /// </summary>
    public Vector2 placementArea;

    /// <summary>
    /// The list of prefabs sample and randomly place
    /// </summary>
    public GameObjectParameter prefabs;

    float m_SeparationDistance = 1f;
    GameObject m_Container;
    GameObjectOneWayCache m_GameObjectOneWayCache;

    /// <inheritdoc/>
    protected override void OnCreate()
    {
        m_Container = new GameObject("Foreground Objects");
        m_Container.transform.parent = scenario.transform;
        m_GameObjectOneWayCache = new GameObjectOneWayCache(
            m_Container.transform, prefabs.categories.Select(element => element.Item1).ToArray());
        m_SeparationDistance = CalculateMaxSeparationDistance();
    }

    /// <summary>
    /// Generates a foreground layer of objects at the start of each scenario iteration
    /// </summary>
    protected override void OnIterationStart()
    {
        var seed = SamplerState.NextRandomState();
        var placementSamples = PoissonDiskSampling.GenerateSamples(
            placementArea.x, placementArea.y, m_SeparationDistance, seed);
        var offset = new Vector3(placementArea.x, placementArea.y, 0f) * -0.5f;
        foreach (var sample in placementSamples)
        {
            var instance = m_GameObjectOneWayCache.GetOrInstantiate(prefabs.Sample());
            instance.transform.position = new Vector3(sample.x, sample.y, depth) + offset;
        }
        placementSamples.Dispose();
    }

    /// <summary>
    /// Deletes generated foreground objects after each scenario iteration is complete
    /// </summary>
    protected override void OnIterationEnd()
    {
        m_GameObjectOneWayCache.ResetAllObjects();
    }

    /// <summary>
    /// Calculates the max separation distance needed between placed objects to be sure that no two objects will overlap
    /// </summary>
    /// <returns>The max separation distance</returns>
    float CalculateMaxSeparationDistance()
    {
        var maxPlacementDist = 0f;
        foreach (var category in prefabs.categories)
        {
            var prefab = category.Item1;
            var size = prefab.GetComponent<Renderer>().bounds.size * maxScaleFactor;
            var maxDiagonal = Mathf.Sqrt(size.x * size.x + size.y * size.y + size.z * size.z);
            maxPlacementDist = Mathf.Max(maxPlacementDist, maxDiagonal);
        }
        return maxPlacementDist;
    }
}

[lessw2020]

Awesome, thanks @sleal-unity!