Error in MRR computation

[Ricocotam]

Hi, You compute the MRR metric using the following code :

  if 'MRR' in metrics_map:
    score = 0.0
    for rank, item in enumerate(pred):
      if item in gt:
        score = 1.0 / (rank + 1.0)
        break
    out[metrics_map.index('MRR')] = score

  if 'MRR@10' in metrics_map:
    score = 0.0
    for rank, item in enumerate(pred[:10]):
      if item in gt:
        score = 1.0 / (rank + 1.0)
        break
out[metrics_map.index('MRR@10')] = score

But according to wikipedia it should be more like :

  if 'MRR' in metrics_map:
    score = 0.0
    for rank, item in enumerate(pred):
      if item in gt:
        score += 1.0 / (rank + 1.0)
    out[metrics_map.index('MRR')] = score / len(pred)

  if 'MRR@10' in metrics_map:
    score = 0.0
    for rank, item in enumerate(pred[:10]):
      if item in gt:
        score += 1.0 / (rank + 1.0)
out[metrics_map.index('MRR@10')] = score / 10

Note the score += and score / len(pred)

[Ricocotam]

Is misread...