search

Jun-CEN / Open-world-3D-semantic-segmentation

[ECCV 2022] Open-world Semantic Segmentation for LIDAR Point Clouds
69 stars 10 forks source link

REAL vs Upperbound #5

Closed luoyuchenmlcv closed 2 years ago

luoyuchenmlcv commented 2 years ago

Thanks for sharing your great work!

I am indeed curious about the remarkable result in incremental learning, in your paper, even you use pseudo labeling instead of ground truth to fine-tune the model from M_o to M_c, REAL is even better than its theoretical upper bound, which is even better than using ground truth. Could you explain why is the counter-intuitive result? mIoU mIoUnovel mIoUold
Closed-set 58.7 0 78.3
Upper bound 73.8 62.5 77.6
Finetune 0 0 0
Feature extraction 5.5 2.1 6.6 LwF 6.1 2.4 7.3
REAL 74.9 62.2 79.1

Jun-CEN commented 2 years ago

Hi,

Thanks for your interest. First, the performance of our method is closed to the upper bound is understandable, as original model already has good performance on the old classes and the ground truth of new class is given. The pseudo labeling preserves the knowledge of old classes very efficiently. Second, using the ground truth of all classes may not be the true upper bound of incremental learning. There is no guarantee that learning one by one must be worse than learning once for all classes. Learning one by one does have potential to be better than learning once for all classes, but nowadays incremental learning still cannot beat training from scratch using all labels in most research area, especially in image recognition, where old samples are missing or partly missing during incremental learning. But in semantic segmentation, the old samples naturally exist, as the whole point cloud contains lots of points of old classes.

I hope my explanation is helpful.

Best regards, Jun

luoyuchenmlcv commented 2 years ago

Thanks for your quick response!

But using the prediction to replace old sample's ground truth and beat ground truth still makes me feel confused, could this mean that your pseudo label prediction for old samples has already been good enough, well ALIGNED with old sample's distribution in val set, and in contrast, the unused old samples' ground truth has larger semantic shift with old sample's distribution in val set?

Jun-CEN commented 2 years ago

It's hard to say whether your opinion is correct or not. From my point of view, I think the performance of REAL and upper bound is comparable, not clearly better, if you refer to the test set results. Our results can definitely show that the pseudo labels are very good choice to alleviate catastrophic forgetting, but it's hard to say whether it is better than ground truth or not.