Enable evals on multiple databases for OpenAI and Anthropic generators

defog-ai / sql-eval

Evaluate the accuracy of LLM generated outputs

Apache License 2.0

520 stars 56 forks source link

Closed rishsriv closed 2 months ago

rishsriv commented 2 months ago

In addition:

pass through join hints to the OpenAI and Anthropic generator prompts when c=0. These do not make any significant difference to performance (the difference in +- 0.5% with and without for gpt-4o, gpt-4, and claude-3.5-sonnet), but we are passing them through for the sake of apples to apples comparisons with other models.
remove glossary from all prompts, as it is not needed

wendy-aw commented 2 months ago

Thanks for incorporating translations in the openai/anthropic runners! Just a few comments from me.

rishsriv commented 2 months ago

Fixed, thanks Wendy!