Review: Doc Recall Eval

deepset-ai / haystack

:mag: AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.

Apache License 2.0

16.96k stars 1.85k forks source link

The MRR evaluator always returns 0.0 when we index/retrieve chunks of the original document.

Related to this: https://github.com/deepset-ai/haystack/blob/main/haystack/components/evaluators/document_mrr.py#L77
Because we index/retrieve not the original document with its full content, but chunks of the original document (the result of applying the DocumentSplitter) - so this will always fail on the if ground_document.content in retrieved_document.content
One possible solution is to switch the test: if retrieved_document.content in ground_document.content, but this is still an expensive operation, and maybe if the meta field of a document contains an unique id, we could compare the ids instead.
Discuss with Thomas Stadelmann about "context matching" in dC

deepset-ai / haystack

Review: Doc Recall Eval #7772