Closed LuckyAnJooo closed 5 months ago
Hi, thanks for using our code.
Palm2 indeed lacks the capability as pred_model or optim_model. The accuracy will drop when the prompt is transfered to Palm2. But I didn't notice such dramatic drop in our experiment, since the human prompt can get 0.43 according to the paper. There may be a couple of reasons: 1. Different palm2 version; or maybe palm2 is updated so the output format doesn't follow the answer extraction format. You may look into the output file of the test.py and check what's the response look like. Since we mostly focus on OpenAI models, there could be answer format or extraction issue for Palm2. 2. There exists randomness, it may happen that palm2 just cannot understand this one prompt. You may check our paper on openreivew: https://openreview.net/pdf?id=22pyNMuIoa. There is more experiment on page 16 section C.2. It shows PaLM2 is a weaker model than GPT3.5 when it is used as the base model in PromptAgent.
Yes, you may take any optimized prompt to test. The agent defaultly pick the highest average reward path.
The same as question 1. PaLM2 doesn't has a good capability of following complex instructions, proved in the paper. Also there may be randomness or format issue. Since it has been 5 month after the experiment, the model or api maybe updated but not the PromptAgent code accordingly. (Actually not only the PaLM, GPT3.5 and 4 are also updating. You may use gpt-4-turbo to save some money : ) .
Thanks for your patient response.
Hello, I appreciate your kind words and efforts to replicate our results on my computer. Here is the process I followed: First, I installed the necessary environment as per the "Installation" instructions. Then, I ran the main function using the command "python src/main.py --task_name bigbench...". Afterward, I selected the prompt from "best_reward_path_selected_node" in the "data.json" file for testing. I placed that specific prompt in the "prompt.txt" file and used the following script for testing:
I conducted tests with both 'gpt-3.5-turbo' and 'palm2', yielding results of 0.7468 and 0.1266, respectively. Here are my questions:
I look forward to your response.