sweep-ai[bot]
commented
1 year ago Here's the PR! https://github.com/wwzeng1/gpt-prompt-engineer/pull/3.
To create this ticket, I used gpt-4. You have 38 gpt-4 tickets left.
- Install Sweep Configs: Pull Request
Step 1: 🔍 Code Search
I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.
Some code snippets I looked at (click to expand). If some file is missing from here, you can mention the path in the ticket description.
https://github.com/wwzeng1/gpt-prompt-engineer/blob/df5230f6f51b728e34a07d90c3ef0bb8dc01d91c/README.md#L1-L114 https://github.com/wwzeng1/gpt-prompt-engineer/blob/df5230f6f51b728e34a07d90c3ef0bb8dc01d91c/gpt_prompt_engineer_Classification_Version.ipynb#L1-L416 https://github.com/wwzeng1/gpt-prompt-engineer/blob/df5230f6f51b728e34a07d90c3ef0bb8dc01d91c/gpt_prompt_engineer.ipynb#L1-L394Step 2: 🧐 Snippet Analysis
From looking through the relevant snippets, I decided to make the following modifications:
| File Path | Proposed Changes |
|---|---|
gpt_prompt_engineer.ipynb |
- Modify the generate_optimal_prompt function to implement the iterative approach. After generating the initial prompts and identifying errors, add a loop where the model rewrites the prompt based on the errors and retests them. |
| - Add a new function to combine different prompts. This function should take a list of prompts as input and return a new prompt that is a combination of the input prompts. | |
| - Modify the error handling to ask the model to explain its reasoning when an error occurs. This could be done by adding a new function that takes the erroneous prompt and test case as input and returns the model's explanation. | |
gpt_prompt_engineer_Classification_Version.ipynb |
- Similar modifications as in gpt_prompt_engineer.ipynb need to be made in this notebook as well. Modify the generate_optimal_prompt function to implement the iterative approach, add a new function to combine different prompts, and modify the error handling to ask the model to explain its reasoning when an error occurs. |
Step 3: 📝 Planning
I have created a plan for writing the pull request. I am now working my plan and coding the required changes to address this issue. Here is the planned pull request:
Implement iterative approach and error handling improvements
sweep/iterative_approach
Description
This PR implements an iterative approach to improve the prompt generation in the
gpt-prompt-engineerproject. It also includes improvements to the error handling when a generated prompt gives an error on a test label.Changes Made
- Modified the
generate_optimal_promptfunction ingpt_prompt_engineer.ipynbto implement the iterative approach. After generating the initial prompts and identifying errors, a loop is added where the model rewrites the prompt based on the errors and retests them.- Added a new function to combine different prompts in
gpt_prompt_engineer.ipynb. This function takes a list of prompts as input and returns a new prompt that is a combination of the input prompts.- Modified the error handling in
gpt_prompt_engineer.ipynbto ask the model to explain its reasoning when an error occurs. This is done by adding a new function that takes the erroneous prompt and test case as input and returns the model's explanation.- Made similar modifications in
gpt_prompt_engineer_Classification_Version.ipynbto implement the iterative approach, add the function to combine prompts, and modify the error handling.Checklist
- [ ] Tested the changes locally
- [ ] Updated the documentation
- [ ] Added unit tests
- [ ] Verified the code follows the project's style guidelines
- [ ] Created and reviewed the PR
Related Issue
Link to the related issue: Issue Link
Step 4: ⌨️ Coding
I have finished coding the issue. I am now reviewing it for completeness.
Step 5: 🔁 Code Review
Success! 🚀
I'm a bot that handles simple bugs and feature requests but I might make mistakes. Please be kind! Join Our Discord
Let's say a generated prompt gives an error on a test label.
What I sometimes do manually is I ask GPT4 to explain its reasoning, and argue that the correct label should have been FOO.
When it concedes that it was wrong, I then ask it to rewrite the prompt so that it would have gotten this test label correct.
That would be another approach, that is iterative, rather than just randomly picking 10 prompts.
This could be extended further, a little bit based upon the idea of genetic algorithms:
Take the initial prompt, run it, find the errors. Pick 10 random errors and ask it to rewrite the prompt based upon each. Ask it to combine different prompts.