search

xuxw98 / ESAM

EmbodiedSAM: Online Segment Any 3D Thing in Real Time
https://xuxw98.github.io/ESAM/
181 stars 13 forks source link

About the result shown on tab1 #4

Closed mellody11 closed 1 week ago

mellody11 commented 1 week ago

Hi, thank you for opening your great work!

I would like to further confirm whether the dataset used for the experiment in tab1 is scannetV2 or scannet200.

Because in paper SAI3D: Segment Any Instance in 3D Scenes, they report their 30.8 AP result on the ScannetV2 dataset.

Thank you for your attention.

XXXCARREY commented 1 week ago

Thanks for your interest! We initially referenced the first version of the SAI3D paper, which reported performance tested on ScanNet200, and we directly used those results. Only recently did we discover the issue that they actually used ScanNetV2 for their tests. We re-evaluated their method on ScanNet200 and found that the performance had slightly decreased, but the difference was not significant. Therefore, we did not modify the performance of SAI3D in Table 1, even though it is slightly higher than the actual value.

mellody11 commented 1 week ago

However other methods (e.g. HDBSCAN Nunes et al.) also seem to experiment on Scannet, not Scannet200.

xuxw98 commented 1 week ago

Hi, thanks for pointing out this issue!

In our Table1, we refer to the v1 version of SAI3D on arxiv and the performance of HDBSCAN, Nunes et al., Felzenszwalb et al. and UnScene3D are reported from the SAI3D paper. They are actually evaluated on the 18 classes of ScanNet and so the actual performance of them on ScanNet200 should be lower.

We are evaluating ESAM on the 18 classes of ScanNet in class-agnostic manner and compare it with the above mentioned methods. We will record these experiments in a new table (class-agnostic on ScanNet). As for our Table1, we will try to implement the first four baselines on ScanNet200.

The new experiments will be soon updated in this issue. We will also revise the paper accordingly.

mellody11 commented 1 week ago

I would like to express my respect for your rigorous academic attitude.

Thank you for your reply.

xuxw98 commented 1 week ago

The class-agnostic 3D instance segmentation performance of ESAM-E on ScanNet (18 classes) is:

Method Type VFM AP AP50 AP25
SAI3D offline Semantic-SAM 30.8 50.5 70.6
ESAM-E online FastSAM 49.3 71.4 85.8