LzVv123456
commented
11 months ago Your thought is indeed the case. When the pretrained model is too weak, you can not expect the frozen model to generate good key prototypes and hence it is hard to include the correct prompt into the candidate prompt group using KNN (see the variation of baselines under different pre-trained weights in Table2). As a remedy, you can try to increase the query range, say top-10, to increase the chance at the cost of the inference speed. This can be viewed as an inherent property of this framework, it is built upon a strong pretrained model. Otherwise, one can not even expect the prompt-tuning technique itself to be useful.
vanity1129
Hi, @LzVv123456
I compared the Top-3 accuracy without prompt (95.64%) and the accuracy with Prompt (94%) under same setting. According to my understanding, retrieving Top-3 Key_protos when there is no Prompt is also equivalent to finding the corresponding candidate Prompt, and thus the corresponding Value_protos can be subsequently used for classification. In terms of the two classification accuracies, the ability of the candidate Prompt to contain the Target Prompt is actually largely determined by the pre-trained network itself. Then, if the pre-trained network itself is weak when retrieving the candidate Prompt in this step, it will also directly affect the result. I hope to hear your detailed insights about this part. Looking forward to answering questions at your leisure!