search

Unity-Technologies / com.unity.perception

Perception toolkit for sim2real training and validation in Unity
Other
926 stars 177 forks source link

Labeling of trees #160

Closed HenrikWerner closed 1 year ago

HenrikWerner commented 3 years ago

Is there any way to improve the labeling of the leaves of trees? Labeling and leaves do not exactly match and sometimes several leaves are not labeled at all (and then they get the label of the object behind them).

mkamalza commented 3 years ago

Hi Henrik. In order for us to help you, we would need a bit more info about the issue. Can you perhaps provide some screenshots of the visualized bounding boxes, as well as how your labeling is set up for the trees?

HenrikWerner commented 3 years ago

Thanks for your quick reply. Sure, here are a few pictures. Behind the tree are buildings and the sky. I use the SemanticSegmentationLabeler. Config: Trees are labeled green, buildings in the background gray. As you can see, some leaves on the tree are not getting the correct label / color assignment. They just get the color / label of the objects that are behind them and overall the correspondence between leaves and label / color is not very accurate 1 2 3

JonathanHUnity commented 3 years ago

There appear to be two issues at play here.

The first is that the perception package do not currently support transparency for in ground truth rendering. This causes the leaf sections to be show up as large geometric shapes instead of individual leaves. Is it important to you to have pixel-perfect segmentation of leaves?

The second is that the geometry in the semantic segmentation does not line up with the positions of the actual leaves. My hunch is that the tree has a geometry or vertex shader which is deforming the tree. Vertex and geometry shaders are also not run when generating ground truth. What kind of tree is this? Speed tree or some other asset store package? You may find an option to disable tree movement which will get the geometry to be static and line up properly.

HenrikWerner commented 3 years ago

Just for clarification, the result shown in the images above is the same as in the generated segmentation images.

It could be important for the training to have pixel-perfect segmentation of leaves or at least no misclassified parts. It is an asset store package, but it just uses the HDRP/Lit-Shader and there is no tree movement at all.

I think I found the reason: The first image is without Alpha-Clipping and the second without Double-Sided. Alpha clipping explains why it does not match the shape and double-sided why some parts are not labeled at all. Without both, the labeling is exactly the same, but of course it looks very bad. A B

JonathanHUnity commented 3 years ago

OK, very interesting. We haven't played with Alpha-Clipping or Double-Sided. Could you tell us which asset store package you used so we can test ourselves?

Querying the alpha channel of textures is on our backlog but we do not yet have an ETA. In the meantime if you have some experience with shaders you may want to try playing with the two shaders used for labeling:

https://github.com/Unity-Technologies/com.unity.perception/blob/master/com.unity.perception/Runtime/GroundTruth/Resources/SemanticSegmentation.shader

https://github.com/Unity-Technologies/com.unity.perception/blob/master/com.unity.perception/Runtime/GroundTruth/Resources/InstanceSegmentation.shader

We will update this thread when we have a solution available to try.

HenrikWerner commented 3 years ago

You can test it with any object that changes it appearance (especially the shape / contour), if you deactivate alpha-clipping or double-sided. I tested it beside other assets with trees from the Forest Environment - Dynamic Nature from NatureManufacture in the asset store. But as I said, it also happens with other assets.

HenrikWerner commented 3 years ago

I had the time to look at it myself. I changed the shader, but this was not enough. I also changed the SemanticSegmentationCrossPipelinePass so that the texture of each material is accessible by the shader. In the image you can see the result. Do you want me to contribute this somehow (it fits my application, but it definitely still has to be tested carefully)? Tree

JonathanHUnity commented 3 years ago

By the way, we have a branch with experimental support for transparency here: https://github.com/Unity-Technologies/com.unity.perception/tree/seg_opacity Feel free to give feedback if you try it out!

totti0223 commented 3 years ago

SemanticSegmentationCrossPipelinePas

Hi, @HenrikWerner, I'm also interested in labeling leaves. Would you mind sharing your modification of CrossPipelinePass if possible?

gespona commented 2 years ago

Hi ! I'm working with trees also. Support for transparency helped here but still facing the issue with double-side. Could you share or point me how to change to make visible double-side ?

gespona commented 2 years ago

Hi ! any help with double-side please?

aryan-mann commented 1 year ago

Hey all! The Perception team just released 🎉 Perception 1.0 🎉, a major update to the toolset! We have added support for transparency in our segmentation pass which should enable the above use-case. Let us know how it goes! Closing this issue for now.

There are a bunch of cool features to check out such as ray tracing, depth, a new output format SOLO, normals, and more! Here is the full list of changes: Perception 1.0 Changelog