Closed Ahnsun closed 2 years ago
Hi, the config retinanet_clip_r101_fpn_1x_coco.py
is used to train the RetinaNet model with the CLIP ResNet-101 backbone. You should modify the dimensions and backbone architectures if you want to use the config to train the ViT-B model. Besides, it is difficult to directly use the ViT-B model on detection tasks due to the quadratic complexity of self-attention and the large image sizes used in detection. Therefore, we didn't test the ViT-B model on COCO in our paper.
Get it, thanks a lot!
Hi, I used the pre_trained model ViT-B-16.pt and the config retinanet_clip_r101_fpn_1x_coco.py. However, the embed_dim of CLIPTextContextEncoder is 1024 while the embeded_dim of pretrained model is 512.