Getting metric value for each user

[ajaykv1]

Hi, i am currently working on a project where I am trying to compare different algorithms together to see if the results are statistically significant. How can I get the HR, NDCG and MRR values for each user instead of getting one value for the results.

For example, if my dataset has 10 users in the test set, how can I retrieve all the 10 MRR,NDCG and HR values for each user instead of an average. As of right now, the model is returning "INFO test result: OrderedDict([('recall@10', 0.0233), ('mrr@10', 0.0098), ('ndcg@10', 0.0125), ('hit@10', 0.025), ('precision@10', 0.0025)])". But, I want each metric for all the users in the dataset.

Please let me know how to get that.

Thank you!

[ajaykv1]

Is there an update on this issue?

[Wicknight]

Hello @ajaykv1 , sorry for the late reply. We reuse RecBole's implementation in the evaluation module. So you need to modify some of your RecBole code to do this, or rewrite it under RecBole-CDR. Take the example of modifying RecBole code:

1. You first need to find the metrics that you want to calculate for each user, take Hit@10 for example.
2. Locate the Hit class in RecBole/recbole/evaluator/metrics.py.
3. Then you can find metric_info function in this file, the return value here is metric result for each user.

   def metric_info(self, pos_index):
       result = np.cumsum(pos_index, axis=1)
       return (result > 0).astype(int)