Why freeze_plm is false?

[preke]

I tried several tutorial codes for my own data and found that once change the freeze_plm to be True in

prompt_model = PromptForClassification(plm=plm,template=mytemplate, verbalizer=myverbalizer, freeze_plm=True)

the performance would degrade a lot. So, am I doing something incorrectly?

Or is that if we set the freeze_plm to be False, it means we actually are fine-tuning the pre-trained models with the designed prompts?

[ningding97]

freeze_plm = True means that all the parameters of the model will be frozen (fixed) during training, and only the parameters of prompts would be updated. This will degrade the performance because there are fewer parameters being optimized. But if the model is super-large, the performance may not be significantly affected. See this paper. If freeze_plm = False, it means the parameters of the model will also be optimized during the process.

[preke]

Thanks for your timely reply! I also want to know if I am correct: once we set freeze_plm=False: we have two situations below:

1. If the template and the verbalizer are manually set, we only update the parameters in the PLM (as no additional parameters are introduced in the prompts) by the "auto-completing loss". Actually, I am not sure if it is similar to the prompt-based fine tuning in Section 4 of the LM-BFF paper (Making Pre-trained Language Models Better Few-shot Learners).

2. If we introduce parameters in the prompts (we hope to learn better prompts), the parameters in the prompts and PLM are updated simultaneously during training.

Thanks again for your time!

[ningding97]

Yes! Your understanding is correct. Actually the prompt-based fine-tuning is kind of a de facto standard now, and it could be traced back to earlier papers.