Questions about AUCEvaluator

[lzhbrian]

Hi, I have a few questions about the AUCEvaluator.

When I configure different topN to the same recommender, AUCEvaluator gives different results, so I looked into the implementation of AUCEvaluator and I get a bit confused...

Why do we need to give topN when we calculate AUC? I mean normally, AUC for ranking problems in recommender systems is defined as: which has nothing to do with topN

I find myself hard to understand the code. Is there a reference or something for that implementation that I could look into?

Thanks in advance !

[KeqiangWang]

Yes，just for AUC you are right. But for top-n evaluator in librec, i.e., AUC@n, is not AUC. AUC@n is used for top-n ranking list. Another measure, MAP@n is also different from MAP. If you want to calulate the AUC, you may implement it by yourself now as follows: . And consider the limited memory, I suggest you calculate the every user's auc and get the average value.

[lzhbrian]

Thanks for the reply @KeqiangWang So is there any plan for official maintainers of Librec to implement this AUC I have mentioned for ranking tasks? I think its more common for people to use AUC instead of AUC@K when evaluating implicit feedback tasks? (such as in the paper of BPR, SBPR, ...)

[KeqiangWang]

The major problem is that if we want to compute every user's AUC, we need to get all ranking item list for all users in the librec framework. However, the needed memory is too large(10000 users and 10w items may need larger than 30G memory). So for the sake of simplicity, we just store the top-n list. Another way is just get user's ranking item list and get its AUC one by one, than get the average value. But this way is not suitable for recommender framework.

[lzhbrian]

@KeqiangWang Thanks! I will figure that out by myself.