Synthetic Dataset "guess_correctness"

eth-sri / llmprivacy

MIT License

38 stars 4 forks source link

Synthetic Dataset "guess_correctness" #2

Closed redwyd closed 6 months ago

redwyd commented 6 months ago

Could you tell me the meaning of "model_free_eval" and "model_aided_eval" fields of "guess_correctness" in synthetic dataset?

mveroe commented 6 months ago

"model_free_eval" contains the evaluation of the guess correctness based solely on manual checks, while "model_aided_eval" contains the evaluations corrected by GPT4. src/synthetic/eval_tools.py should give you some more insight in the details of how these two methods work.