:mag: AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
Some datasets contain document relevance scores or at least a sorted order of ground truth relevant documents, for example HotpotQA. In order to evaluate rankers and retrievers regarding their ability to not only distinguish relevant from non-relevant documents but also to sort relevant documents by their relevance, we need a new component DocumentNDCGEvaluator. The currently available DocumentMRREvaluator considers relevancy of documents to be binary.
Some datasets contain document relevance scores or at least a sorted order of ground truth relevant documents, for example HotpotQA. In order to evaluate rankers and retrievers regarding their ability to not only distinguish relevant from non-relevant documents but also to sort relevant documents by their relevance, we need a new component DocumentNDCGEvaluator. The currently available DocumentMRREvaluator considers relevancy of documents to be binary.