search

defog-ai / sqlcoder

SoTA LLM for converting natural language questions to SQL queries
Apache License 2.0
3.42k stars 218 forks source link

spider dataset evaluation #12

Closed chuangchuang-wang closed 1 year ago

chuangchuang-wang commented 1 year ago

Has anyone tested the effect on the spider dataset? I have tested the result and the accuracy rate is only 30%, is it normal

rishsriv commented 1 year ago

Hi there, the Spider dataset uses SQLite while our model is trained to generate PostgreSQL. Additionally, our model is optimized for enterprise use-cases when columns descriptions are well defined. You can read more about the differences in the Spider dataset and our SQL-Eval here: https://defog.ai/blog/open-sourcing-sqleval/