search

ArvinZhuang / DSI-QG

The official repository for "Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation", Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei, Ming Gong, Guido Zuccon and Daxin Jiang.
MIT License
114 stars 19 forks source link

How to calculate the metric HITS #14

Open oussaidene opened 8 months ago

oussaidene commented 8 months ago

Hi, I am a bit confused on how the metric HITS is calculated. According to the article, HITS is defined as "the proportion of the correct docids ranked in the top 1 and top 10 predictions." However, it appears that the code implements HITS@N differently.

In the code, HITS@N is calculated as the proportion of queries that retrieve at least one correct document in the top N rankings. Could someone confirm if my understanding is correct?

hi-i-m-GTooth commented 7 months ago

Hi @oussaidene ,

Refer to Lightning AI, the hit rate is 1.0 if there is at least one relevant document among all the top k retrieved documents. I think the definition of the hit rate should not contain the ranking concept. Hence, the calculation in the code should be correct. Hope this helps!