search

meidachen / STPLS3D

🔥 Synthetic and real-world 2d/3d dataset for semantic and instance segmentation (BMVC 2022 Oral)
236 stars 20 forks source link

SoftGroup for STPLS3D #16

Open thangvubk opened 2 years ago

thangvubk commented 2 years ago

Hi @meidachen,

Thank you for sharing a new interesting outdoor dataset for instance segmentation. Based on your shared code, we have implemented our SoftGroup for STPLS3D, which achieves AP/AP50/AP25 of 46.4/62.0/69.5 on val set, significantly higher than the existing ones. I hope this may serve as a solid baseline for Urban3D@ECCV2022 challenge. Feel free to check https://github.com/thangvubk/SoftGroup.

meidachen commented 2 years ago

Hi @thangvubk ,

Really appreciate your feedback and implementing SoftGroup for STPLS3D!!!

Regarding the performance, we have recently found an issue in the evaluation script we provided in HAIS as well as the CodaLab evaluation. I just fixed it an hour ago and found out that our HAIS baseline was incorrect (the HIAS performance should be 35.1% instead of the 40.4% that is currently reported in the paper and our website). I'm not sure if you are using the same script for evaluation or not, but if you are, could you please re-evaluate it using the updated script?

The bug was at line 226 where I mistakenly used //1 instead of //1000, which made all our previous results incorrect.

Same thing for the codaLab evaluation script at here where I mistakenly used //1 instead of //10000

I'm working to make the correction for results on all our pages, paper, and codalab evaluation server, etc.

I'm not sure if this will affect your result performance but just want to provide all the information for your consideration.

If possible, could you please share the entire result table with me? I can put it on this GitHub as a baseline and point other people to the SoftGroup repo for implementation details.

Thank you again!! and sorry for any inconvenience, this may be caused.

thangvubk commented 2 years ago

We use our implementation from ScanNet v2 and we modified it to support different datasets. I only set the min_npointfor evaluation of STPLS3D to 10. We use 1000 to get the ids.

https://github.com/thangvubk/SoftGroup/blob/a87940664d0af38a3f55cdac46f86fc83b029ecc/softgroup/evaluation/instance_eval.py#L255.

I am not sure whether anything else need to be taken into consideration. Feel free to have a look.

thangvubk commented 2 years ago

When we change 1000 to 1. The AP/AP50/AP25 are 53.5/68.8/77.7

meidachen commented 2 years ago

Wow!! really nice work!! Could you please share the result table with me? and I will make the update here.

thangvubk commented 2 years ago

Here it is image

meidachen commented 2 years ago

Great! Thank you!

meidachen commented 2 years ago

SoftGroup result was uploaded to this repo with all related information. Thanks again for the implementation, and I hope to see you in the Urban3D workshop soon!