search

defog-ai / sql-eval

Evaluate the accuracy of LLM generated outputs
Apache License 2.0
448 stars 47 forks source link

Clarified instructions for restaurants database #183

Closed rishsriv closed 2 weeks ago

rishsriv commented 2 weeks ago

Previously, we had instructions like this:

Match city_name and food_type case-insensitively. Match with ILIKE and percent sign for substring matching for all other string matches.

This was confusing because both these lines were saying the same thing. I have changed this to Match all strings case-insensitively for all database types.

wendy-aw commented 2 weeks ago

Ok noted! Quick question, do you think we should keep the percent wildcard sign instruction since that hasn't got anything to do with case insensitivity?

rishsriv commented 2 weeks ago

Good point will fix post lunch!

rishsriv commented 2 weeks ago

Thanks for the clarifications!

I'm good with matching all strings case-insensitively for now. We should specify more granular and challenging instructions in the future (sp because our current benchmark is getting a bit saturated now with scores in/approaching 90+%). But that's a task for the future and not the best use of our time right now, methinks!

Let me know. I have modified the instructions to Match all strings case-insensitively using wildcard operators for clarity

wendy-aw commented 2 weeks ago

Okie gotcha!