search

truera / trulens

Evaluation and Tracking for LLM Experiments
https://www.trulens.org/
MIT License
1.84k stars 163 forks source link

fix groundedness with no supporting evidence #1193

Closed nicoloboschi closed 3 weeks ago

nicoloboschi commented 3 weeks ago

Items to add to release announcement:

Currently if you use feedback groundedness_measure_with_cot_reasons and one of the hypothesis is returning no supporting evidence, the feedback is completely broken and not even the score is visible. This is due to the fact this feedback implies to always have a "reason" behind the score.

:rocket: This description was created by Ellipsis for commit 1c9a72915db6391a341824c6ad5b7fea5d88c4dd

Summary:

This PR fixes an issue in the groundedness_measure_with_cot_reasons feedback function to handle cases with no supporting evidence without breaking.

Key points:

Generated with :heart: by ellipsis.dev

epinzur commented 3 weeks ago

i wonder if when running in deferred mode, this change would prevent trulens from seeing this as a failed evaluation and then would not re-run it.

nicoloboschi commented 3 weeks ago

i wonder if when running in deferred mode, this change would prevent trulens from seeing this as a failed evaluation and then would not re-run it.

It might be as side effect. Anyway all other feedback providers don't fail if the reason can't be found so I think it's better to align to the others' behaviour