brandenchan
commented
3 years ago Hi @sophgit, cool to see you're already using the new evaluation nodes!
So I do expect a difference in Retriever eval stats when run using Retriever.eval() vs a EvalRetriever node but actually I would have expected Retriever.eval() to do worse. The reason is that Retriever.eval() is doing a Closed Domain retrieval evaluation, meaning that a successful retrieval is based on whether the right document is retrieved (this is judged using the document's ID). The EvalRetriever node is actually an Open Domain retrieval. A document is considered correctly retrieved as long as the answer string is contained within the retrieved document, and we don't check its ID.
Open Domain retrieval is easier than Closed Domain so I actually would have expected the opposite results to what you posted. Down the line, we will want to implement Closed Domain in the Eval nodes.
For now, we should definitely look into this problem to see what is going wrong. Currently it seems like you are using a different dataset to what is in the tutorial, which makes this a little bit tricky. But to start, maybe you could help us out by looking a little more closely at the Retriever's predictions in each case.
For the Retriever only case, could you try adding the return_preds=True argument?
i.e. res = retriever.eval(return_preds=True)
res is now a dictionary and res["predictions"] returns the predictions of the retriever.
For the Evaluation Nodes case, add debug=True when initializing the EvalRetriever
i.e. eval_retriever = EvalRetriever(debug=True)
After running the evaluation, eval_retriever.log will return all the retriever's predictions.
If you are able to compare the results from each and see which samples differ, that could be really informative for us!
sophgit
Timoeller
Question Hey :) I have been evaluating my pipeline with your updated evaluation tutorial in Colab. Great tutorial, thanks!! However, I noticed that the retriever recall is different when evaluating it separately from the retriever recall in the pipeline evaluation. Shouldn't the value be the same? Top_k_retriever was set to 5 both times. When doing the same evaluation with the older tutorial (with Finder instead of pipeline) the retriever recall is the same in both the single evaluation and finder evaluation.
Additional context the result of
retriever_eval_results = retriever.eval(top_k=5, label_index=label_index, doc_index=doc_index)is:For 255 out of 272 questions (93.75%), the answer was in the top-5 candidate passages selected by the retriever. Retriever Recall: 0.9375 Retriever Mean Avg Precision: 0.8526960784313726
the result of
p = Pipeline() p.add_node(component=retriever, name="ESRetriever", inputs=["Query"]) p.add_node(component=eval_retriever, name="EvalRetriever", inputs=["ESRetriever"]) p.add_node(component=reader, name="QAReader", inputs=["EvalRetriever"]) p.add_node(component=eval_reader, name="EvalReader", inputs=["QAReader"]) results = []for q, l in q_to_l_dict.items(): res = p.run( query=q, top_k_retriever=5, labels=l, top_k_reader=3, index=doc_index, ) results.append(res)is:Retriever
recall: 0.9265 (252 / 272)
Retriever (Speed)
No indexing performed via Retriever.run() Queries Performed: 272 Query time: 3.9433569139969222s 0.014497635713223978 seconds per query
Is there a reason why the recall deteriorates in the pipeline? Thank you!