semantic segmentation is not enabled in `apartment_0000` and `kujiale_0000` scenes

[RachithP]

The actors and static mesh components in apartment_0000 and kujiale_0000 scenes do not have semantic tags as part of their tags. These are required to enable semantic segmentation in these scenes. Also, the programmatic way to process these semantic tags, and convert them to custom depth stencil value is missing.

How to reproduce:

• run generate_images.py with render pass value set to segmentation. The output images will be blank.

[mikeroberts3000]

We are already requiring the artist to group actors into top-level folders in the World Outliner view, based on their semantic categories. Using folders in the World Outliner view is convenient for editing and debugging, but the folder structure is not available at runtime, so we need to somehow convert the folder structure to tags, and ultimately to custom depth-stencil values.

We could require the artist to do this manually, but that seems error-prone. Instead, we should automate this tagging procedure via a Python script that can be invoked via the command-line, the Unreal console, or via a custom editor button. In principle, this Python script could also be used to set custom depth-stencil values, so we wouldn't need to do it at runtime.

[RachithP]

I agree. Since we cannot obtain the World Outliner view folder structure in runtime, and allowing this process to be manual is error-prone, we can use Python APIs to perform this task. This way we can execute this script on a need-basis, whenever new actors/components are added, and are re-arranged.

Some references:

• https://docs.unrealengine.com/5.2/en-US/PythonAPI/class/Actor.html#unreal.Actor.get_folder_path
• https://docs.unrealengine.com/5.2/en-US/PythonAPI/class/_ObjectBase.html#unreal._ObjectBase.set_editor_property
• https://docs.unrealengine.com/5.2/en-US/PythonAPI/class/StaticMeshComponent.html#unreal-staticmeshcomponent

I've a pseudo code here. One drawback in this code is that we cannot specify different semantic tags for the components under the same actor. This could be a problem for actors that have components merged, yet each component needs to have different semantic tags.

import unreal

editor_subsytem = unreal.get_editor_subsystem(unreal.EditorActorSubsystem)
level_actors = editor_subsystem.get_all_level_actors()
for actor in level_actors:
    # get semantic tags 
    actor_path = actor.get_folder_path()
    semantic_tag = actor_path.split('_')[-1]

    # get all staticmeshcomponents
    static_mesh_components = actor.get_components_by_class(unreal.StaticMeshComponent)

    # apply semantic tags to all components
    for component in static_mesh_components:
        component.set_editor_property("component_tags", [semantic_tag])

[mikeroberts3000]

This sketch is useful, thank you for that.

• I don't think that pseudocode will correctly handle the case where the semantic category name has underscores in it, e.g., pillow_case or something.
• I think our design philosophy should be to provide sensible default behavior that doesn't require any additional artist steps most of the time, but allows for full artist control in cases when the artist wants it. For example, in the sketch above, we are unconditionally setting component tags. I think it would be better to only set the component tag if the component doesn't have a spear:semantic:whatever tag already set. This would make the most common case (i.e., an actor with a single semantic class) very easy for the artist, because they could just place the actor in the 00_whatever folder in the World Outliner. But in rare cases when an artist wants additional control, they can always express exactly what they want by manually tagging individual components.