search

confident-ai / deepeval

The LLM Evaluation Framework
https://docs.confident-ai.com/
Apache License 2.0
3.34k stars 260 forks source link

Retrieval context removed #309

Closed hguhlich closed 11 months ago

hguhlich commented 11 months ago

The retrieval context is no longer part of the LLMTestCase as of this commit. Is this intended? I think, it makes a lot of sense to also test for the quality of the retrieved context.

If you really don't want to have that anymore, you should at least remove it from the documentation: https://docs.confident-ai.com/docs/evaluation-test-cases#retrieval-context

penguine-ip commented 11 months ago

Hey @hguhlich, it was removed because there are currently no metrics supporting ranking similarity, and also we're no evidence so far it is being used by users. Are you using it? Happy to add it back for our release early next week.

hguhlich commented 11 months ago

Shouldn't the Contextual Relevancy Metric from RAGAS use the retrieval context? From your documentation:

Remember, context is the ideal retrieval results for a given input and typically come from your evaluation dataset, whereas retrieval_context is your LLM application's actual retrieval results.

So, comparing the context with the input doesn't make sense if the context should be relevant by definition. We should compare the retrieved context with the input.

penguine-ip commented 11 months ago

Definitely something I overlooked, thanks for taking the time to look through the commit history and pointing it out! The new release on Monday will contain the fixes. Also come join our discord if you're on there, would love to have you in the community: https://discord.com/invite/a3K9c8GRGt

penguine-ip commented 11 months ago

@hguhlich https://docs.confident-ai.com/docs/evaluation-metrics#ragas, its out, thanks!