search

scene-verse / SceneVerse

Official implementation of ECCV24 paper "SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding"
https://scene-verse.github.io
MIT License
200 stars 3 forks source link

The preprocess code or coordinate transformations for HM3D, ScanNet and Structured3D #21

Closed MrZihan closed 2 months ago

MrZihan commented 3 months ago

Dear Authors,

We sincerely appreciate your outstanding work! However, the lack of preprocessing code for HM3D, ScanNet, and Structured3D, as well as the coordinate transformations between SceneVerse's 3D scenes and the original scenes in these datasets makes it difficult for future researchers to effectively utilize your annotated data with the Habitat simulator (HM3D) or RGB-D images (ScanNet, Structured3D). Please forgive my intrusion. Sincerely hope you can provide the relevant assistance. Thank you and your team for the outstanding contributions to the community!

Buzz-Beater commented 3 months ago

We have already provided an alignment example in preprocess/, preprocessing for other scene point clouds should be similar, can you try following the instructions to preprocess other datasets as well and let me know if any more instructions or scripts should be provided? Thx :-)

MrZihan commented 3 months ago

Okay, thank you very much! I will attempt to make the modifications. If all the datasets are processed using similar code, hope the coordinate transformations will align accurately. 🙏

MrZihan commented 3 months ago

We have already provided an alignment example in preprocess/, preprocessing for other scene point clouds should be similar, can you try following the instructions to preprocess other datasets as well and let me know if any more instructions or scripts should be provided? Thx :-)

Thank you! Referring to vil3dref, I have successfully aligned the coordinate system of the ScanNet dataset. However, when working with the HM3D dataset, I found that the script you provided does not align the HM3D instance ID with the SceneVerse instance ID. As for Structured3D, I haven't yet found a way to handle the coordinate system. I would greatly appreciate your help, especially in verifying the accuracy of the HM3D preprocessing script.

Buzz-Beater commented 2 months ago

Hi, we provided the HM3D id match using sceneverse2hmsemantic.py for aligning IDs. As for Structured3D, can you specify a bit more about what is the problem? Thx :-)

MrZihan commented 2 months ago

Hi, we provided the HM3D id match using sceneverse2hmsemantic.py for aligning IDs. As for Structured3D, can you specify a bit more about what is the problem? Thx :-)

The sceneverse2hmsemantic.py seems to have some errors. In the generated tgt_id2obj_id.json, the Sceneverse IDs and HM3D IDs do not correspond well. For example, in HM3D, an object might be classified as a chair, but the corresponding Sceneverse ID identifies it as a cup. The IDs is completely scrambled. Could the use of set() in line 45 of sceneverse2hmsemantic.py caused the order to become unpredictable?

yixchen commented 2 months ago

Hi, indeed sceneverse2hmsemantic.py may not reproduce the correct mappings for different OS and Python versions. Thanks for pointing out. We have released the mappings in HM3D_tgtID2objID.zip. Hope these address your concerns.

MrZihan commented 2 months ago

Hi, indeed sceneverse2hmsemantic.py may not reproduce the correct mappings for different OS and Python versions. Thanks for pointing out. We have released the mappings in HM3D_tgtID2objID.zip. Hope these address your concerns.

Thanks a lot for your help! It works now!