search

defog-ai / sqlcoder

SoTA LLM for converting natural language questions to SQL queries
Apache License 2.0
3.27k stars 205 forks source link

benchmarking question #83

Closed lalehsg closed 6 months ago

lalehsg commented 6 months ago

Hi all, I am wondering if there's any comparison between Defog and Mixtral 8x7B Instruct model has been done. Thanks in advance!

rishsriv commented 6 months ago

Hi there! You can run a comparison on sql-eval, with the code below. IIRC, mistral-medium via the Mistral API is the 8x7B model. On SQL-Eval, it is 63% accurate, compared to 90% for sqlcoder-7b-2

python -W ignore main.py \
  -db postgres \
  -o "results/results.csv" \
  -g mistral \
  -f "prompts/prompt_mistral.md" \
  -m mistral-medium \
  -p 5