Best way to implement "max quantity of detectable objects per image" and "one of" controls for randomization?

[lessw2020]

For the work I'm doing, I'd like to be able to implement two specific controls related to the amount and type of items per randomized image. Thus, wanted to see if there are any recommendations for how to implement the following types of controls to the randomization for the detectable objects:

1 - A "one of" feature. Example - I have diagnostic test prefabs that show different results. I'd like to be able to have a "one of" selected from each test type, randomly chosen per image. Thus I may have Covid test, model A, and select to show "one of" type model A, but randomly selecting from the model A prefabs that show positive-1, positive-2/negative/invalid, etc. I only want to show a single prefab of this type, not simply have a pct chance that of each various version appearing.

2 - max or explicit quantity per image of detectable objects - in some cases I want to generate images that have only a single detectable item displayed, selected from all the available prefabs.
Another dataset might need only 2 types of items to detect shown, others 3, etc. Is there a way to script or otherwise control this (beyond the make a new project, only add one item, or two items). I'd like to work within a much larger scene and have dynamic control of this.

[sleal-unity]

Hi lessw2020,

I may need some additional clarification and context for your two questions:

1. Is this a conditional probability problem? Let me know if this is right: you have multiple models (A, B, C) and each model type has a set of result permutations (positive-1, positive-2/negative/invalid, etc.). In one frame, you want to select a model type, say model A, then randomly place objects of only that type (model A) but with different results.

2. You can likely create a custom randomizer that can configure this max-quantity property for you. Randomizers don't actually have to "randomize" any particular property. You can use them to simply spawn X number of objects per frame if that is what you are looking for. Randomizers have simulation lifecycle hooks that make it easy to program simple spawning operations like this. I would recommend phase 2 of the perception tutorial to explore how to make a custom randomizer. I would also recommend that you take a look at how the ForegroundPlacementRandomizer included with the perception package is implemented to get an example of how you could create a custom object spawner for your use case. Let me know if I'm missing the point of your question though.

[lessw2020]

Hi @sleal-unity, Thanks for the help! 1 - It's basically an exclusivity problem based on the ability to organize subsets of objects into clusters. (that said the conditional probability option you described is a different problem but something I would also like to be able to do). Let me try and provide a crisper example of this 'one of' though for now: I have test model type A. Test A has 5 prefabs, each of which show a different test result for model Test A ( 3 types of positive, a negative, and invalid result) but are otherwise all Test A models. (same thing for Test B, C, etc. but just to focus on A...) Thus, on a given frame, I want to be able to say "randomly place one of test A", and have only one of the 5 permutations of test A selected and placed.
That extrapolates to having one of test A, one of test B, etc. but for example just focusing on test A and placing only one of it's permutations is the express problem. I guess really it's a question of whether/how prefabs can be grouped in the interface, or in code, to form subsets/clusters where you can randomly request only 1 from that subset at a time, and have multiple subsets that provide a more hierarchical organization to the prefabs during generation. Currently in the tutorial it's just has the concept of 'all objects are independent and equal' and a single probability of selection applied to all of them equally. In looking at the Foreground placement randomizer (thanks for the link) I'm guessing I just need to write some code to clump the prefabs into their respective groupings and then select it. Since the concept of "one of" is frequently used (i.e. most obj detection augmentation pipelines ala albumentations, etc. have a built in "one of" function) wasn't sure if there was a prebuilt way to do this.

2 - max-quantity - thanks for noting that randomizers can also simply be spawners. Will use that and write a max quantity spawner. The code for the foregroundplacement btw answered a lot of questions on this beyond just the max qty. (example, I have to wrap the object in a game object to handle global and then local rotation, but I see in the code now how to instantiate a game object programmatically so also answers that).

Thanks again!

[sleal-unity]

No problem, happy to help!

I have a few suggestions on how to organize your code to support your first point. However, this advice is contingent on integrating the latest changes to the perception package into your project instead of the current release you get by default from the package manager. Here is a doc that explains how to add the perception package to your project after you clone the latest master branch to disk. The next release should be out within a week or so, but installing the perception package through this method will let you take advantage of our latest features.

That being said...

First, I would create a new ScriptableObject cluster class that enables you to create assets that specify a group of prefabs in the Unity Editor. I have a PR in the works (Edit: it's in master now) that will make this class simpler to write, but here's an example of said class that works with the current master branch of the perception package:

using UnityEngine;
using UnityEngine.Perception.Randomization.Samplers;

[CreateAssetMenu(fileName="NewCluster", menuName="Test/Cluster")]
public class Cluster : ScriptableObject
{
    /// <summary>
    /// Specifies a array of prefabs that define this cluster
    /// </summary>
    public GameObject[] clusterPrefabs;

    /// <summary>
    /// Returns a uniformly sampled random prefab from the cluster
    /// </summary>
    public GameObject Sample()
    {
        var sampler = new UniformSampler();
        return clusterPrefabs[(int)(sampler.Sample() * clusterPrefabs.Length)];
    }
}

After defining this class, go to your project window and right click anywhere in the window. In the context menu, select Test -> Cluster to create a new cluster asset. If you click on the newly created asset, in its inspector window, there should be a GameObject array field that you can add a list of prefabs to in order to define a new type of cluster.

Next, define a new categorical cluster parameter class for randomly selecting one type of cluster from the list of all defined cluster types. Here's an example of this cluster parameter class:

using System;
using UnityEngine.Perception.Randomization.Parameters;

/// <summary>
/// A categorical parameter for randomly selecting clusters
/// </summary>
[Serializable]
public class ClusterParameter : CategoricalParameter<Cluster> {}

Finally, create a custom randomizer to select a random cluster and prefab from that cluster. Here's an example on how to do that:

using System;
using UnityEngine;
using UnityEngine.Perception.Randomization.Randomizers;

[Serializable]
[AddRandomizerMenu("Test/Cluster Randomizer")]
public class ClusterRandomizer : Randomizer
{
    /// <summary>
    /// The list of clusters to choose from
    /// </summary>
    public ClusterParameter clusters;

    protected override void OnIterationStart()
    {
        // Select a random cluster asset
        var cluster = clusters.Sample();

        // Select a random prefab from the selected cluster
        var prefab = cluster.Sample();

        // Write code to instantiate instances of this prefab here...
    }
}

In your scenario's inspector window, add the new cluster randomizer. In the ClusterRandomizer's cluster parameter, add your new cluster assets to the categorical list. This randomizer should now have all the information it needs to randomly sample prefabs from different clusters.

[sleal-unity]

Alternatively, you can write the Cluster ScriptableObject using a GameObjectParameter. This will simplify the code a bit and give you more helpful UI options for customizing the list of prefabs included in your Cluster.

using UnityEngine;
using UnityEngine.Perception.Randomization.Parameters;

[CreateAssetMenu(fileName="NewCluster", menuName="Test/Cluster")]
public class Cluster : ScriptableObject
{
    public GameObjectParameter clusterPrefabs;
}

Though you will need to make a custom editor for the Cluster ScriptableObject to allow the GameObjectParameter field to render properly in a Cluster asset's inspector window. This is an artifact of how Unity uses the old IMGUI framework to render editors by default, though categorical parameters like the GameObjectParameter class have UIs authored using the new UI Elements framework. Below is an example of how to leverage the ParameterUIElementsEditor abstract class provided by the perception package to quickly write a custom editor for your ScriptableObject that is compatible with all perception parameter classes.

using UnityEditor;
using UnityEditor.Perception.Randomization;

[CustomEditor(typeof(Cluster))]
public class ClusterEditor : ParameterUIElementsEditor { }

Note: Any editor scripts must be placed inside an "Editor" folder within your project. "Editor" is a special folder name in Unity that prevents editor code from compiling into a player during the build process. For example, the file path for the ClusterEditor script above could be ".../Assets/Scripts/Editor/ClusterEditor".

Sorry for the wall of text! Let me know if you need any additional clarification on anything.

[lessw2020]

Hi @sleal-unity, Thanks a ton for the code above! I will implement this next week and really appreciate the help (and not at all a wall of text, rather lots of useful and appreciated info!) Also, happy to note that I just finished training on 100% synthetic generated covid rdts, and as a test, skipped the real world image fine tuning and jumped directly to real world images for detection. (we are waiting for the real images to drop with this new diagnostic). Anyway, detection performance was outstanding.
Lots more to do like the above to improve our synthetic generation, but happy to be able to show that for medical diagnostics there is a direct transfer from Unity generated data to real world diagnostic images. And thanks for all the help getting to this accomplishment. Thanks again and will implement all the above next week so much appreciated.

[sleal-unity]

No problem. Hit me up if you get stuck anywhere. By the way, your project sounds awesome! Glad the perception package is working for ya!