Closed Swiftyos closed 4 months ago
add to the prisma parser tests the example here
Converted to draft until I have a good before and after benchmark.
@majdyz @ntindle @Torantulino @aarushik93 Please all feel free to commit to this PR any tweaks to the prompts (or target a pr into this one)
I will update the benchmark system then try and test if the prompts are improving or not
For the record: I approved this merge, the results should be better in reality, but it's quicker for us to find out for ourselves than wait for the new system to be ready which will give us objective evidence.
Once this system is ready, we'll go through many iterative rounds of prompt refinement based on what we can measure.
Looks potentially positive, but it's very hard to say with prompts. There's not much I can do to review this.
How can we objectively know whether it's better?
It would be very helpful to see the evidence you have so far @Swiftyos.