search

defog-ai / sql-eval

Evaluate the accuracy of LLM generated outputs
Apache License 2.0
540 stars 57 forks source link

huggingface evaluation dataset not found #134

Closed minjunp closed 4 months ago

minjunp commented 5 months ago

Hi!

First of all, thank you for sharing your amazing work. I have two questions:

  1. In https://huggingface.co/defog/llama-3-sqlcoder-8b?fbclid=IwZXh0bgNhZW0CMTEAAR28ltksnyUgl8zct50HTLk2F0PBrcnoCD2IGkj2ARhwT-RYcAbkPNcwyQQ_aem_ARWYpBMyD_hPULx94U2gVeIhhzf8gcjRXqW80H7cTR5_IGFKYnE-CIUwSvx2yM7WlgQThhTkUeY4i1SO5CBoyPse I see sqleval-classic-Scores and sqleval-instruct Scores. I see that your model outperformed GPT-4 turbo, and I was wondering if you released the dataset used for evaluation.

  2. In your prompt, I see {instructions} part. Are they like full_instructions column in https://github.com/defog-ai/sql-eval/blob/main/data/instruct_advanced_postgres.csv? Or do also only use db schema as input?

Thank you so much!

wongjingping commented 4 months ago

Hi @minjunp, you can find the dataset in this same repo, in the data folder. To run the eval, you can follow the commands here.

Are they like full_instructions column in https://github.com/defog-ai/sql-eval/blob/main/data/instruct_advanced_postgres.csv?

Yes, although we currently just use the instructions column in that same csv. The difference is that the instructions column contain only the relevant instructions required, while the full_instructions column contain a few other irrelevant / distracting instructions. Instructions are an optional addition to the db schema (which is always required).