search

minghanqin / LangSplat

Official implementation of the paper "LangSplat: 3D Language Gaussian Splatting" [CVPR2024 Highlight]
https://langsplat.github.io/
Other
670 stars 70 forks source link

Ground-truth data for Evaluation #21

Closed yashbhalgat closed 8 months ago

yashbhalgat commented 8 months ago

Hi @minghanqin , this is an interesting work and the performance is impressive! :slightly_smiling_face:

In the paper you mention that:

"we extend the LERF dataset by annotating ground truth masks for textual queries, enabling the evaluation of the open-vocabulary 3D semantic segmentation on the LERF dataset....." "Therefore, we further manually annotated additional challenging localization samples to better evaluate method performance."

Could you please share your newly annotated dataset (and if possible, the eval code too), so that it's possible to have a fair comparison with the LangSplat method?

Thank you! Yash

minghanqin commented 8 months ago

You can download the newly annotated dataset for the LERF dataset from the following link: https://drive.google.com/file/d/1QF1Po5p5DwTjFHu6tnTeYs_G0egMVmHt/view?usp=sharing.

Regarding the evaluation code, we are in the process of organizing and preparing the codebase for public release. It requires some time to ensure that the code is well-documented and user-friendly. We plan to make it available on GitHub soon.

yashbhalgat commented 8 months ago

Thanks @minghanqin! I had a couple of follow-up questions:

  1. In Table 1 of the LangSplat paper (https://arxiv.org/pdf/2312.16084.pdf) where you report Localization accuracy -- do you consider all the queries that LERF considered (around 75 in total) in addition to the new queries that you include in your extended dataset?
  2. In Tables 1 and 2, I noticed that you do not include the "Bouquet" scene, even though it was a part of the LERF paper. Could you please let me know the reason for this?
minghanqin commented 8 months ago

Hi, Thank you for your detailed inquiries regarding our LangSplat paper. I'm happy to provide clarification on the points you've raised:

  1. Regarding the Localization accuracy in Table 1, we did overlap with most of the queries that LERF considered but did not include all. 2.We intended to include the "Bouquet" scene in our evaluations. However, we encountered technical difficulties with running COLMAP on this particular scene.
yashbhalgat commented 8 months ago

Thanks very much for the clarification! @minghanqin :)