A question about the process of P-tunning-v2.

[QWTforGithub]

Thank you for your well-organized code. Since my major si computer vision, I am not familiar with NLP. I'm very interested in P-Tunning.
1) In P-tuning-v2, do I need to take the Prompts when training Pre-trained Model? Or just take the Prompts when I train a downstream task.
2) For the initiation of a prompt:
prompts = torch.arrange()
prompts = torch.nn.Embeding(prompts) Is it normal to use the above initialization method？ 3) For training a downstream task, I need to freeze all of the Model' parameters, but not the Prompts' parameters, right? 4) For Prompt Deep, I need to reinitialize the Prompts on each level, right ( Same initialization as 2) ) ? I'm sorry for asking so many questions. Looking forward to your reply.

[Xiao9905]

Hi @QWTforGithub ,

1. Not necessary. The prompts are only necessary in the downstream transfer learning. Though, pre-training with prompts is an interesting direction to explore.
2. Yes. This is the most simple initialization method. Some researchers propose advanced initialization (such as in Lester et al. 2021 and Gu et al. 2021) approach, but they are for P-Tuning v1 (or namely the shallow prompt tuning).
3. Yes.
4. Yes.

Hope these could be helpful. Plz feel free to ask any additional questions.

[QWTforGithub]

Hi @QWTforGithub ,

1. Not necessary. The prompts are only necessary in the downstream transfer learning. Though, pre-training with prompts is an interesting direction to explore.
2. Yes. This is the most simple initialization method. Some researchers propose advanced initialization (such as in Lester et al. 2021 and Gu et al. 2021) approach, but they are for P-Tuning v1 (or namely the shallow prompt tuning).
3. Yes.
4. Yes.

Hope these could be helpful. Plz feel free to ask any additional questions.

Thank you very much！

[QWTforGithub]

Hi @QWTforGithub ,

1. Not necessary. The prompts are only necessary in the downstream transfer learning. Though, pre-training with prompts is an interesting direction to explore.
2. Yes. This is the most simple initialization method. Some researchers propose advanced initialization (such as in Lester et al. 2021 and Gu et al. 2021) approach, but they are for P-Tuning v1 (or namely the shallow prompt tuning).
3. Yes.
4. Yes.

Hope these could be helpful. Plz feel free to ask any additional questions.

Thank you for your reply. Prompts need to make Position Encoding? I see your code doesn't have one.

[Xiao9905]

@QWTforGithub ,

If you use P-Tuning v2, the positional encoding is not necessary, as prompts are trainable and can cover the fixed positional encodings in the training. In our experiment, we do not see the difference between with and without positional encodings.

[QWTforGithub]

@QWTforGithub ,

If you use P-Tuning v2, the positional encoding is not necessary, as prompts are trainable and can cover the fixed positional encodings in the training. In our experiment, we do not see the difference between with and without positional encodings.

I got it. Thank you very much!