search

drprojects / superpoint_transformer

Official PyTorch implementation of Superpoint Transformer introduced in [ICCV'23] "Efficient 3D Semantic Segmentation with Superpoint Transformer" and SuperCluster introduced in [3DV'24 Oral] "Scalable 3D Panoptic Segmentation As Superpoint Graph Clustering"
MIT License
508 stars 65 forks source link

Question about results on s3dis without colors #118

Closed gardiens closed 1 month ago

gardiens commented 1 month ago

Hello, I am currently attempting to reproduce the results presented in your paper regarding the S3DIS dataset without using color information. According to your findings, the omission of color data resulted in a mean Intersection over Union (mIOU) decrease of only 2.5 points. Did you try to conduct this experiment with either the pc_tiling or xy_tiling methods activated.

Due to my computing machine, I need to activate this tiling my mIOU dropped approximately of 10 points, yielding an mIOU around 60, Is this relatively drop expected? While I understand that the absence of radiometric features would naturally lead to a decline in mIOU, I am unsure about the expected magnitude of this impact. Sincerely, Pierrick

drprojects commented 1 month ago

Hi @gardiens, indeed removing radiometric information typically results in a performance drop.

As you noticed, the performance drop we communicate when removing RGB from pointwise features on S3DIS 6-fold is near 2.7 mIoU. However, this specific ablation only removes colors at training time, but still uses the same partition as the other ablations in the table. Said otherwise, the RGB color was used when computing the partition but not used as a point attribute when training. I agree this is a bit misleading.

Yet, in a previous paper where we worked on mutimodal learning on S3DIS, we did run the no-RGB ablation with a MinkowskiNet backbone on S3DIS folds 2 and 5 (https://arxiv.org/pdf/2204.07548). We did observe performance drops close to -10 mIoU, which seems to align with what you are observing too. Besides, it is possible that SPT suffers more from the lack of radiometric information than MinkowskiNet, since the partition construction then only relies on local, handcrafted features which may fail to capture all nuances of the local geometries.

Best, Damien

gardiens commented 1 month ago

Hi @gardiens, indeed removing radiometric information typically results in a performance drop.

As you noticed, the performance drop we communicate when removing RGB from pointwise features on S3DIS 6-fold is near 2.7 mIoU. However, this specific ablation only removes colors at training time, but still uses the same partition as the other ablations in the table. Said otherwise, the RGB color was used when computing the partition but not used as a point attribute when training. I agree this is a bit misleading.

Yet, in a previous paper where we worked on mutimodal learning on S3DIS, we did run the no-RGB ablation with a MinkowskiNet backbone on S3DIS folds 2 and 5 (https://arxiv.org/pdf/2204.07548). We did observe performance drops close to -10 mIoU, which seems to align with what you are observing too. Besides, it is possible that SPT suffers more from the lack of radiometric information than MinkowskiNet, since the partition construction then only relies on local, handcrafted features which may fail to capture all nuances of the local geometries.

Best, Damien

Thank you for your quick reply!