search

CVMI-Lab / PLA

(CVPR 2023) PLA: Language-Driven Open-Vocabulary 3D Scene Understanding & (CVPR2024) RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding
Apache License 2.0
262 stars 11 forks source link

scene captions? #11

Closed jwyndreee closed 1 year ago

jwyndreee commented 1 year ago

Dear authors, thank you for your great work.

I was skimming through the code and found out that scene captions are never encountered in caption_head.py.

I also noticed that the caption_idx directory of your shared onedrive does not contain matching indices of scenes.

Do you plan to release them later?

Thanks,

Dingry commented 1 year ago

Hi, scene caption processing can be found here: https://github.com/CVMI-Lab/PLA/blob/main/pcseg/models/head/caption_head.py#L63. Scene caption doesn't need a corresponding index since it relates to the whole scene.

jwyndreee commented 1 year ago

thanks!

jwyndreee commented 1 year ago

Sorry for bothering again, the code you referred is the exact point where I have found that scene captions are never encountered in caption_head.py.

I searched further and found out that on your onedrive /PLA/data/s3dis/text_embed, the caption_scene file of s3dis does not exist, and instead is named as that of scannet.

image

Is it okay if we just rename it to s3dis and use it?

Thanks in advance!

jwyndreee commented 1 year ago

Renaming it to s3dis and just using it results in error. There are certain scenes, e.g. 'Area_3_storage_2', without scene captions.

Dingry commented 1 year ago

Sorry for the wrong name for this file. We have corrected it and fixed the error. You can try it now. Those scene names don't have scene captions because they lack raw 2D images.

jwyndreee commented 1 year ago

https://github.com/CVMI-Lab/PLA/blob/7ba735f6c382622205cfb52a96a3b4eba3e42c51/pcseg/models/head/caption_head.py#L84

pooled_objs to pooled_feats?

Dingry commented 1 year ago

Yes. Thx for pointing this out.