search

Jun-CEN / Open-world-3D-semantic-segmentation

[ECCV 2022] Open-world Semantic Segmentation for LIDAR Point Clouds
67 stars 10 forks source link

Class Indices for Unseen Classes #13

Closed Yang-Li-2000 closed 1 year ago

Yang-Li-2000 commented 1 year ago

In the repo "semantic_kitti_api-master", it seems that the class index for unseen classes is 5 after mapping, which corresponds to class 13, 16, 20, 99, 256, 257, and 259 before mapping.

image Screen Shot 2022-11-08 at 17 03 57

In val_cylinder_asym_ood.py in the repo "Open_world_3D_semantic_segmentation", in contrast, there does not seem to be a class for unseen objects. It looks like a few classifiers (self.logits2 in forward_dummy_final in segmentator_3d_asymm_spconv.py) are specialized for producing uncertainty scores for unseen classes. However, it is not clear which class corresponds to the unseen class.

image

Additionally, in Section 5.1 of the paper, novel classes are defined to be {barrier, construction-vehicle, tra ffi c-cone, trailer}.

image

In semantic-kitti.yaml in both of the repos, however, there does not seem to exist barrier, construction-vehicle, tra ffi c-cone, or trailer.

I am confused about how to tell which points belong to the unseen class when running val_cylinder_asym_ood.py.

Jun-CEN commented 1 year ago

Hi,

(1) The val_cylinder_asym_ood.py. is to generate and store the uncertainty score for every point. After that, you should use our semantic_kitti_api or nuScenes_api to evaluate the open-set performance based on the stored uncertainty files. The unknown class for open-set evaluation is at https://github.com/Jun-CEN/semantic_kitti_api/blob/bd918e4b3f143d1f12132b5b5e586abf5f0c7149/auxiliary/np_ioueval.py#L86-L87 (2) K^{sk} is the unknown class for SemanticKITTI dataset, and K^{ns} is the unknown class for nuScenes dataset. So for SemanticKITTI, the unknown class is only other-vehicle.

Yang-Li-2000 commented 1 year ago

I am still confused about the class indices for unseen objects when using semantic kitti dataset. What are the differences between class 5 and class 20 in voxel_label_origin in train_cylinder_asym_ood_final.py?

Jun-CEN commented 1 year ago

Hi,

The unknown class in SemanticKITTI is only 5. The 20 in train_cylinder_asym_ood_final.py is the position of our dummy classifier. In other words, the probability of unknown class is at the 20th position of the logits. loss_dummy means the second largest probability should be the unknown class, which refers to the model calibration technique in the paper.