Semantic segmentation results do not show void classes(color is black)

[jmwang0117]

appreciating your excellent work !!! I want to follow your work, but when I used your most recent code in the replica room_0, I discovered that the void class could not display in the image,but the generated sem.mp4 can display void class normally. Why is this? I recall using your code a few months ago, and the output image showed a void class(color is black). Please give me some advice on how to switch back to the old code. Thanks a lot!!!

[Harry-Zhi]

Hi @jmwang0117 ,

Actually we did not change the image rendering results as well as visualisation.

The images shown in tensorboard and mp4 video are masked using the void regions: https://github.com/Harry-Zhi/semantic_nerf/blob/2608d2fdb66137e08b34b02864e56dd866b877f0/SSR/training/trainer.py#L1010

For individual images and labels rendered in function _renderpath (https://github.com/Harry-Zhi/semantic_nerf/blob/2608d2fdb66137e08b34b02864e56dd866b877f0/SSR/training/trainer.py#L1151), we did not apply the voild masking yet. If you need it, you could modify the codes within this function correspondingly.

Hope it helps.

[jmwang0117]

Thank you for your reply ! Is this related to moving the label in the code, as the visualization of the void class always produced some puzzling issues in other scenarios? I tested it in room 2 where the void class was divided into chairs and ceilings, as I found out. In your paper, did you add a mask to each void class?

[Harry-Zhi]

Hi @jmwang0117 ,

Sorry for very late reply.

In the paper, yes I masked out the void regions as I did not penalised on void classes so the network predictions on these void regions could be arbitraty. For visualisation purposes, I make them black using GT void masks and make them black.

For C semantic classes, the MLP predict labels with a range of [0, C-1], In the code snippets you cropped above,
valid colour is obtained by slicing off the first element of 'colour_map_np' whose first colour element corresponds to void region, i.e., black colour.

To generate labels with void regions masked, you could refer to masking void regions in label. Another way to do so in rendering function is to increate the label you predict by 1, and directly apply complete color map here

The comments there may be a bit out-of-dated and could raise some confusion. I will clean these comments soon.

[Harry-Zhi]

Close it for now. Feel free to re-open it if you have any further questions.