Closed Daydaylight closed 2 months ago
In general yes, fine-tuning typically overspecializes the model. We explore this and solutions in these papers and associated repos: https://arxiv.org/abs/2109.01903 https://arxiv.org/abs/2208.05592 https://github.com/mlfoundations/wise-ft https://github.com/mlfoundations/patching
After fine-tuning on a small dataset containing new categories (e.g., containing 500 black cats and 500 orange cats), the accuracy of retrieving these two color cats goes up, so does the accuracy of the fine-tuned CLIP to retrieve the original categories go down (e.g., to retrieve the other animal, dog)?