[AGE-273] [sub-issue] Evaluation fails when correct_answer is not set in the test set

Agenta-AI / agenta

The all-in-one LLM developer platform: prompt management, evaluation, human feedback, and deployment all in one place.

http://www.agenta.ai

MIT License

990 stars 163 forks source link

[AGE-273] [sub-issue] Evaluation fails when correct_answer is not set in the test set #1276

Closed mmabrouk closed 2 days ago

mmabrouk commented 5 months ago

To reproduce:

Create a test set without a correct_answer column
Run an evaluation

The evaluation will fail and the result will not be seen

Expected behavior: The evaluation will fail, opening the results will show the llm app output but not the evaluator results (at least the ones that require a correct_answer)

_{From SyncLinear.com | AGE-273}

mmabrouk commented 1 month ago

Todo:

[ ] Check whether this is still relevant
[ ]

mmabrouk commented 5 days ago

Issue reproduced