search

facebookresearch / KILT

Library for Knowledge Intensive Language Tasks
MIT License
894 stars 90 forks source link

How can one output be used for recall@k? #25

Closed mjeensung closed 3 years ago

mjeensung commented 3 years ago

Hi.

I'm using the evaluation script for recall@k on T-REx dataset. I thought it needed the top-k retrieved pages as the outputs of predictions.

However, this line says "guess should provide exactly one output".

https://github.com/facebookresearch/KILT/blob/81d6d267793eec311990a8caac8518d2324a3224/kilt/eval_retrieval.py#L197

I'm curious about how just one output can be used for recall@k? Could you explain what I am misunderstanding regarding how recall@k is computed?

mjeensung commented 3 years ago

I found out that one output could have more than one provenances. I'll close this issue.