search

Unity-Technologies / com.unity.perception

Perception toolkit for sim2real training and validation in Unity
Other
910 stars 177 forks source link

Semantic Segmentation not working (HDRP, 2021.3.19f1) #576

Closed MaximeRoedele closed 1 year ago

MaximeRoedele commented 1 year ago

Hi, I've been working with the perception package for a while now and recently sougth to remake my previous URP simulator in HDRP. I've gone through all of the steps in the official tutorial 3-4 times now, but am not able to generate any semantic segmentations of basic objects in a scene. Would love some help as I am completely stuck 😓

The image below is the current game view in the editor. The cube has a Semantic Segmentation label which is supposed to make it Green, however nothing happens when adjusting the slider labeled Object alpha. The slider named Background alpha, however, renders the entire game-view black. I tested the Semantic Segmentation Labeler by changing the supposed color of the background, but the change does not register either. The label used in the Semantic Segmentation Labeler and on the cube are identical and the Semantic Segmentation Labeler is referenced on the Perception Camera.

I also tried with other Labelers and 2D/3D bounding boxes, but nothing yields any results.

image

Here is a thorough description of the steps i took:

  1. Downloaded the newest recommended version of Unity (2021.3.19f1) as well as the 3D (HDRP) core template before making a new project.
  2. Installed the package through the package manager through Add package from git URL with com.unity.perception. I did not install the tutorial files.
  3. Added the perception camera component to the main camera. Turned off save RGB output as I wouldn't need it and turned the capture trigger mode to manual.
  4. Resolved the errors occuring by turning off Asynchronous Shader Compilation in Edit -> Project Settings -> Editor and changed Lit shader mode to both in the following files:
    • Assets -> Settings -> HDRP High Fidelity
    • Assets -> Settings -> HDRP Balanced
    • Assets -> Settings -> HDRP Performant
    • Assets -> Settings -> HDRPDefaultResources -> HDRenderPipelineAsset There is another file in the last folder: Assets -> Settings -> HDRPDefaultResources -> HDRenderPipelineGlobalSettings with various similar fields, but these only contain the options of Forward or Deferred.
  5. Made a Semantic Segmentation Label Config in the project by Create -> perception -> Semantic Segmentation Label Config. Added a label to this config by pressing Add new label.
  6. Added the label config to the perception camera by pressing the (+) button under Camera labelers and adding a Semantic Segmentation labeler. Linked this to the Semantic Segmentation Labeler described in (5).
  7. Added a cube to the scene and added a Labeling component to it. Added the label from the Semantic Segmentation Labeler described in (5) by pressing Add New Label under Added labels.
StevenBorkman commented 1 year ago

Hey @MaximeRoedele, thanks for using Perception. I haven't got a chance yet to retry what your steps, but have you tried putting the perception camera back to scheduled. Just from your description, there doesn't seem to be anything actually causing the perception camera labelers to activate. Although you do not need the rgb captures. The perception camera still needs to trigger a capture so that the scene can be process and the selected labelers triggered.

I hope this fixes your issue. Steve

MaximeRoedele commented 1 year ago

Hi @StevenBorkman, sorry for the late reply! You were absolutely right, this solved the issue! Thanks :)