Closed ntindle closed 4 months ago
It would probably be a good idea to start measuring the impact of our changes before merging them in.
I suspect we want to be trying to minimise the length of our prompts as much as possible
This should be a great change, but we have to be careful that this doesn't flood the prompt with knock-on error messages and hide the real error which is causing the issue.
How have you found the success rate of this vs the existing error enhancements?
Curious to see some prompts this has constructed.
/benchmark
I suspect we want to be trying to minimise the length of our prompts as much as possible
We certainly want to keep prompts small to avoid the "needle in the haystack" problem, and general degradation of performance that comes with long prompts.
However inserting errors with small documentation snippets like this should in theory be a great use of that token budget in my opinion.
Raise error messages to the prompt