Changing PerceptionCamera.labelers at runtime

[jmacpereira]

We're using a perception camera to label objects that may be spawned by users at runtime, during the dataset collection.

The list of spawnable objects may be very large, so we don't want to add the labels of all of them to the PerceptionCamera.labelers, and would instead prefer to add them dynamically, as they are spawned. We ran into some issues trying this:

1. The API has no methods that directly allow this: CameraLabeler objects can't easily change their LabelEntries, and PerceptionCamera can't easily change its list of labelers - has to be done through RemoveLabeler() and AddLabeler() calls.
2. When trying to remove all labelers and add new ones with additional labels at runtime, it causes slowdown and the Unity game completely hangs. This happens specifically for semantic segmentation.

Perhaps this use case is not considered yet, but it should be possible to add labels to the PerceptionCamera labelers at runtime, as long as no labels are removed - this would make sure semantic segmentation still assigns a unique color to every label.

What do you think? Do you have any other tips that can help our use case?

Thanks

[JonathanHUnity]

Hi @jmacpereira , changing a LabelConfig at runtime is only possible before the config has been used the first time. You can use LabelConfig<T>.Init(IEnumerable<T>) to provide the list of labels. This cannot be changed at runtime, though. The two use cases that we support are to author the label config ahead of time in the editor or create them from code at startup.

You may be able to destroy the GameObject containing the camera and PerceptionCamera and recreate it at runtime. This might not have the same slowdown issue that you see with adding and removing labelers.

[eugeneteoh]

Hi @JonathanHUnity, just wanted to point out that I have a similar use case as well. Having support for dynamically adding labels to the label config during runtime would be great.