search

keirp / automatic_prompt_engineer

MIT License
1.16k stars 149 forks source link

Question about the searching process in Algorithm 1 Line 2-9. #3

Closed li-moxin closed 1 year ago

li-moxin commented 1 year ago

Hi Keiran, thanks for the really great work! I have a question about the implementation of run_instruction_induction.py. As in the paper Algorithm 1 Line 2-9, there is a process of iteratively keeping top k% of prompts and re-evaluating with random training subsets. Maybe I didn't read carefully, is the process implemented in the code? By the way, how to judge the convergence and what is the value of k? Looking forward to your reply~

keirp commented 1 year ago

Hi, this section of the paper refers to an iterative version of APE that alternates between evaluating prompts and generating new candidate prompts that are similar to the best ones generated. This codebase does not include that part of the paper since we found we can get the same performance without the iteration.

li-moxin commented 1 year ago

Thanks a lot :)