search

raoyongming / DenseCLIP

[CVPR 2022] DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting
507 stars 39 forks source link

Open set inference without training? #25

Open Colin97 opened 2 years ago

Colin97 commented 2 years ago

Thanks for the great work. It seems that the released model is trained on the ADE dataset. If we want to test on other text descriptions (classes), we must re-train the model? Is there any other way to do the openest inference without training? For example, can I directly utilize the pre-trained CLIP model to calculate the pixel-wise dot product? Do you have such kind of code support?

Mark-Dou commented 2 years ago

Hello! If I want to re-train the model on the ADE dataset, how can I organize it, I mean whether I need to download it or something else, It is because i can not run the train code successfully. Looking forward to your reply! Thanks a lot!

raoyongming commented 2 years ago

@Colin97 We have tried directly using the pre-trained CLIP models in our early experiments. Unfortunately, we find the locality of the pre-trained CLIP model is not very satisfactory (i.e., the local feature on the feature map may not perfectly reflect the semantic information of the corresponding region/patch). I found that it is necessary to tune the pre-trained parameters on the dense prediction tasks to align the feature map and the input image.

raoyongming commented 2 years ago

@Mark-Dou We have tested the code on our server. We have also seen other folks can run the train and inference code in other GitHub issues. Can you tell us the detailed error occurred in your runs?

Mark-Dou commented 2 years ago

@Mark-Dou We have tested the code on our server. We have also seen other folks can run the train and inference code in other GitHub issues. Can you tell us the detailed error occurred in your runs?

Traceback (most recent call last): File "/home/hanyiwang/.conda/envs/denseclip/lib/python3.8/site-packages/mmcv/utils/registry.py", line 69, in build_from_cfg return obj_cls(**args) File "/home/hanyiwang/.conda/envs/denseclip/lib/python3.8/site-packages/mmseg/datasets/ade.py", line 87, in init super(ADE20KDataset, self).init( File "/home/hanyiwang/.conda/envs/denseclip/lib/python3.8/site-packages/mmseg/datasets/custom.py", line 129, in init self.img_infos = self.load_annotations(self.img_dir, self.img_suffix, File "/home/hanyiwang/.conda/envs/denseclip/lib/python3.8/site-packages/mmseg/datasets/custom.py", line 166, in load_annotations for img in self.file_client.list_dir_or_file( File "/home/hanyiwang/.conda/envs/denseclip/lib/python3.8/site-packages/mmcv/fileio/file_client.py", line 1172, in list_dir_or_file yield from self.client.list_dir_or_file(dir_path, list_dir, list_file, File "/home/hanyiwang/.conda/envs/denseclip/lib/python3.8/site-packages/mmcv/fileio/file_client.py", line 692, in _list_dir_or_file for entry in os.scandir(dir_path): FileNotFoundError: [Errno 2] No such file or directory: 'data/ade/ADEChallengeData2016/images/training'

I am not familiar with mmcv structure, this error means should I download the ADE dataset and put it on the corresponding path? Thanks a lot!

raoyongming commented 2 years ago

@Mark-Dou Yes. Please refer to the mmseg document to prepare the datasets.

zhengyuan-xie commented 1 year ago

Thanks for your great work! I'm a newbie of DL and I wonder where is the code of segmentation loss referred in sec 3.4. It seems that loss_identity and loss_aux in denseclip.py do not fit the description of segmentation loss in paper. Thanks!