Segmentation result display

[MaxEAB]

Hi, Is there a sample code available for overlaying the predicted mask on the image for a visual verification? I am using the pretrained PP_LITE_T_SEG50 model and testing it out on one of the "cityscape" images.

[dagshub[bot]]

Join the discussion on DagsHub!

[BloodAxe]

At this point we haven't made predict() functionality for segmentation models. However it's on the roadmap but work has not been started yet. If you would like to contribute - let us know and we can assist with making a PR for this feature.

[Louis-Dupont]

To add to @BloodAxe, we have a small tutorial that shows how to do it manually. You can also have a look at this documentation page.

[MaxEAB]

Thank you, will take a look at those.

Btw, do you have code available that would help train the PP_LITE_T_SEG50 model on my own custom data?

[Louis-Dupont]

You can have a look at this tutorial: transfer learning - semantic segmentation If you feel like you're still missing some info, don't forget to check this documentation website, it should also include some extra information :)

[MaxEAB]

Great, thank you!

Is this the documentation? -- https://docs.deci.ai/super-gradients/documentation/source/welcome.html

Regrading the overlay: Since the trained model has 19 classes, the mask is of size [1, 19, w, h]. Should I pick up the sigmoid argmax at each pixel location and also threshold at 0.5?

[BloodAxe]

For semantic segmentation you want to apply argmax to outputs (It is irrelevant whether you apply softmax or not) to get hard labels.

[MaxEAB]

Won't I still need a threshold (and thus a sigmoid) for this? Unless "unlabeled background" is one of the 19 classes in the training?

[BloodAxe]

I believe "unlabeled background" is not included as part of model's predictions output. "unlabeled background" usually used to indicate areas that are not annotated and thus not contributing to loss during training. However there could still be valid objects in it, maybe too dark or too complex to annotate. During evaluation, predictions of the model in these regions are ignored and do not contribute to metric.

[MaxEAB]

During evaluation, predictions of the model in these regions are ignored and do not contribute to metric.

What would the predicted mask be for these regions during inference? (I assumed it will get assigned one of the 19 labels, and that I need to use the sigmoid and threshold to weed those out.)

[BloodAxe]

We have upcoming predict() method for segmentation models https://github.com/Deci-AI/super-gradients/pull/1320 And merged PR for visualizing segmentation predictions during training - https://github.com/Deci-AI/super-gradients/pull/1282 This should be available in 3.2