defog-ai / sql-eval

Evaluate the accuracy of LLM generated outputs
Apache License 2.0
485 stars 52 forks source link

Enable argument for specifying how much to round floats to #106

Closed rishsriv closed 3 months ago

rishsriv commented 3 months ago

Earlier, we could get mismatches with rounded floats. For example, in the example below, the 2 frames would not be considered equal

import pandas as pd
from pandas.testing import assert_frame_equal

df1 = pd.DataFrame([[1,2.123,"abc"], [2, 3.123, "xyz"]])
df2 = pd.DataFrame([[1,2.1234,"abc"], [2, 3.1234, "xyz"]])
assert_frame_equal(df1, df2) # throws an error

However, there are instances where we want to specify a float rounding limit beyond which both values are considered equal. In pandas, can do this with df.round(n), where n refers to the number of decimal points. The function takes care of only rounding floats, and not modifying int, bool, object columns

import pandas as pd
from pandas.testing import assert_frame_equal

df1 = pd.DataFrame([[1,2.123,"abc"], [2, 3.123, "xyz"]])
df2 = pd.DataFrame([[1,2.1234,"abc"], [2, 3.1234, "xyz"]])
assert_frame_equal(df1.round(3), df2.round(3)) # runs fine

This PR enabled a new, optional, decimal_points argument for rounding floats. This has been implemented for all runners. Here is an example specifying this for the OpenAI runner

python main.py \
  -db postgres \
  -o results/openai.csv \
  -g oa \
  -f prompts/prompt_openai.md \
  -m gpt-3.5-turbo-0613 \
  -n 10 \
  -p 5 \
  -dp 3