search

defog-ai / sql-eval

Evaluate the accuracy of LLM generated outputs
Apache License 2.0
485 stars 52 forks source link

Added a batch size arg for hf runner #111

Closed rishsriv closed 3 months ago

rishsriv commented 3 months ago

This makes hf runner based evals much faster! To test:

python -W ignore main.py \
  -db postgres \
  -q "data/questions_gen_postgres.csv" \
  -o results/hf_runner_bs8.csv \
  -g hf \
  -f prompts/prompt.md \
  -m meta-llama/Meta-Llama-3-8B-Instruct \
  -b 1 \
  -bs 16