search

ZzZZCHS / Chat-Scene

Code for "Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers" (NeurIPS 2024)
MIT License
90 stars 6 forks source link

Question about the annotations #43

Open KaKa-101 opened 20 hours ago

KaKa-101 commented 20 hours ago

Thanks for your great work! Could you provide the gt_feat_file and gt_img_feat_file? image Currently they are not provided in the Google Drive. Could you also explain how to get the gt_feat_file and gt_img_feat_file? I want to test the model's performance given the GT masks, should I change the following three files to GT files? image image Thanks a lot for your kind help.

QScloud-l commented 14 hours ago

Same issue. Google Drive only contains 4 pt files. The file scannet_val_attributes.pt is required when I attempt to run the evaluation with the code provided in the README, but i don't know how to get it.

ZzZZCHS commented 10 hours ago

Thanks for your great work! Could you provide the gt_feat_file and gt_img_feat_file? image Currently they are not provided in the Google Drive. Could you also explain how to get the gt_feat_file and gt_img_feat_file? I want to test the model's performance given the GT masks, should I change the following three files to GT files? image image Thanks a lot for your kind help.

@KaKa-101 Hi, I've uploaded GT feature files (scannet_gt_uni3d_feats.pt & scannet_gt_videofeats.pt) and GT attribute files (scannet_train_attributes.pt & scannet_val_attributes.pt) to the Google Drive. To get the GT feature files, you can also follow the preprocess readme but change the mask3d masks to the GT masks (which can be found in original ScanNet data).

If you want to test the model using GT masks, you also need to change the annotation files. The current annotation scanrefer_mask3d_train.json is based on mask3d segmented masks. You can create the GT annotation file by transforming the original scanrefer annotation annotations/scanrefer/ScanRefer_filtered_train.json to the same format as scanrefer_mask3d_train.json.

Besides, the model was trained with fixed number of objects (100) for each scene, while GT object numbers are varied for each scene (some of them are even larger than 100), this may lead to a low performance if you directly test the provided model with GT masks. I recommend training it with GT masks before the test.

I'm not sure whether the current code is compatible with GT masks. Some code snippets may need slight modification (for example you need to change this line to assigned_ids = torch.randperm(len(scene_locs))). You can open a new issue if you meet any further problems.

ZzZZCHS commented 10 hours ago

Same issue. Google Drive only contains 4 pt files. The file scannet_val_attributes.pt is required when I attempt to run the evaluation with the code provided in the README, but i don't know how to get it.

@QScloud-l Hi, I've uploaded scannet_val_attributes.pt to Google Drive. I forgot that this file is necessary for evaluation. Sorry for the inconvenience.

QScloud-l commented 9 hours ago

Same issue. Google Drive only contains 4 pt files. The file scannet_val_attributes.pt is required when I attempt to run the evaluation with the code provided in the README, but i don't know how to get it.

@QScloud-l Hi, I've uploaded scannet_val_attributes.pt to Google Drive. I forgot that this file is necessary for evaluation. Sorry for the inconvenience.

Thank you! In fact, I've found the way to generate the scannet_*_attributes.pt files within the preprocess directory. However, I've also found that scan2cap_val_corpus.json is also required for the scan2cap evaluation. I can't find a script to generate this file in the preprocess directory; I've only found some commented-out code. Could you please provide this file or the script? I greatly appreciate your kind assistance.

ZzZZCHS commented 9 hours ago

@QScloud-l Thank you for pointing out the missing files. I've uploaded scan2cap_val_corpus.json just now. Check it~