search

Significant-Gravitas / AutoGPT-Code-Ability

🖥️ AutoGPT's Coding Ability - empowering everyone to build software using AI
MIT License
109 stars 34 forks source link

Prompt refinement #227

Closed Swiftyos closed 4 months ago

Torantulino commented 4 months ago

Looks potentially positive, but it's very hard to say with prompts. There's not much I can do to review this.

How can we objectively know whether it's better?

It would be very helpful to see the evidence you have so far @Swiftyos.

ntindle commented 4 months ago

add to the prisma parser tests the example here

Swiftyos commented 4 months ago

Converted to draft until I have a good before and after benchmark.

@majdyz @ntindle @Torantulino @aarushik93 Please all feel free to commit to this PR any tweaks to the prompts (or target a pr into this one)

I will update the benchmark system then try and test if the prompts are improving or not

Torantulino commented 4 months ago

For the record: I approved this merge, the results should be better in reality, but it's quicker for us to find out for ourselves than wait for the new system to be ready which will give us objective evidence.

Once this system is ready, we'll go through many iterative rounds of prompt refinement based on what we can measure.