search

mlpc-ucsd / MaskCLIP

Code Release for MaskCLIP (ICML 2023)
Other
55 stars 3 forks source link

Could you explain the code of Mask Class Token in the project? #2

Open FYY799 opened 11 months ago

FYY799 commented 11 months ago

Could you explain the code of Mask Class Token in the project?

zh-ding commented 11 months ago

There are two main parts of the Mask Class Token. First one is that we add mask class token (by repeating the class token) to CLIP https://github.com/mlpc-ucsd/MaskCLIP/blob/43049d1d8465b2bfc3785df5d920d87df6e09579/maskclip/modeling/maskclip.py#L165-L168

Second one is that we use the segmentation masks to serve as attention mask in the self-attention layer https://github.com/mlpc-ucsd/MaskCLIP/blob/43049d1d8465b2bfc3785df5d920d87df6e09579/maskclip/modeling/maskclip.py#L50-L66.