Closed base-y closed 2 years ago
Hi, thanks for asking!
Hope these information helpful!
Thanks!
@Timothyxxx thank you. But for Q2, why would the final prefix representations not be the same? Since the gradients of are different when the model is frozen and not frozen? Also for Q1, I see people have extensively spoken about the effectiveness of prompts in PROMPT TUNING (that they work with larger models and low data regime) but I havent seen many people talking about effectiveness of PREFIX TUNING in the same way.
It's easy to understand. Just imagine the first step when loss was propagated into the LM model and prefix-module. They will all be updated if you let the LM unfreeze. But will not change the LM when freezing that.
The efficiency in low-resource is discussed in paper of prefix-tuning. You can refer to that.
@Timothyxxx Thank you for the answers. But for answer 1, I understand the model performance will vary because the LM model is updated (in unfreezed model case) when the loss is propagated backward. But I think the gradient flowing to the prefix representation (prefix learnable parameter) is the same in both the cases (model frozen/unfrozen). In this case, though the entire model performance might be different, the final prefix representations will be the same I guess (since they are updated with the same gradient at every step irrespective of whether the model is frozen or not)?
I still don't understand, I recommend try yourself and compare the weight of two LM (frozen/un-frozen).
@Timothyxxx Hi, I have few queries regarding prefix tuning. Can you shed your opinions on the same since you have worked extensively on it (the github repo of Prefix tuning seems abandoned).