Therock90421
commented
1 day ago Thank you for your question. Actually, you raise an important point. RegionCLIP has good generalization ability and performs well on most zero-shot tasks. However, it mainly provides general knowledge with strong generalization ability, which is often not enough to be fully utilized when applied to small-scale datasets. By pre-training on small-scale datasets, the knowledge of downstream tasks can be injected and the performance is usually improved. In fact, we have also tested skipping pre-training step on different benchmarks. The results show that the performance is not as good as the results with pre-training, and the decline is particularly obvious on datasets that face great domain gap with training data of RegionCLIP (such as FoggyCityscapes).
In the paper, you said that you first pre-train the baseline model with classification loss, regression loss, and adversarial loss (step 2 in the repo). And your model is based on the pretrained RegionCLIP. So what if you skip this pretraining step and train the prompt right away? Because I assume that the pretrained RegionCLIP already has a good feature extractor.