search

mingkaid / rl-prompt

Accompanying repo for the RLPrompt paper
MIT License
286 stars 52 forks source link

Transferring Prompts across LMs #13

Closed 52ie closed 1 year ago

52ie commented 1 year ago

Hi, thanks for your nice work! It seems that the code only supports GPT-2. When will you support RoBERTa which is described in the paper? Or which part of the code should I modify if I want to use RoBERTa as the backbone? Any help is appreciated!

MM-IR commented 1 year ago

Hi, for RoBERTa, it is only for few-shot classification. Our code already supports RoBERTa, you can follow this instruction few-shot classification link.

Hope you are doing well, and feel free to ask any questions! Thanks, since it is a clarification question, I am closing this issue now.

52ie commented 1 year ago

Thanks for your timely reply! I'm surprised at the transferability of the hard prompts shown in "Section 3.3 Transferring Prompts across LMs" and tried to reproduce the results in Figure 2.

I ran run_eval.py with the prompt "AgentMediaGradeOfficials Grade" released for roberta-large on SST-2 at few-shot classification link with different task_lm. The acc I got for roberta-base, roberta-large, gpt2-small, gpt2-medium, gpt2-large are 71.88, 94.17, 49.92, 49.92, 53.21. The performance of the gpt2 series is quite low (nearly random guess).

Can you help me with my reproduction or can you release the prompts you used to get the results in Figure 2? Thanks a lot!

MM-IR commented 1 year ago

Hi, Our Figure is mainly based on 2 tokens. You can try some experiments to see the performance, e.g., I casually take one hand-written prompt like Absolutely (in our qualitative analysis) can transfer very well across LMs.

image

Hope it can help you address your question!