Different AUC implementation with the paper

zhougr1993 / DeepInterestNetwork

1.62k stars 558 forks source link

Different AUC implementation with the paper #71

Closed liuhu-bigeye closed 4 years ago

liuhu-bigeye commented 4 years ago

In the paper, 'AUC' is defined as: $AUC=\frac{\sum_{i=1}^{n} impression_i \times AUC_i}{\sum_{i=1}^{n} impression_i}$ .

However, the two metrics provided in the code are: $GAUC=\frac{\sum_{i=1}^{n}AUC_i}{n}$ and $AUC_{all}$ .

Please correct me if I misunderstand the code.

liuhu-bigeye commented 4 years ago

Reimplementation with the metric in the paper results in a lot higher score, e.g. Amazon Electro: 0.9776. So I guess the AUC defined in the paper is not correct and other works should not use this metric to make fair comparison with the paper.

danni9594 commented 4 years ago

Reimplementation with the metric in the paper results in a lot higher score, e.g. Amazon Electro: 0.9776. So I guess the AUC defined in the paper is not correct and other works should not use this metric to make fair comparison with the paper.

because in their test set, each user only has one positive sample and one negative sample, the number of impressions for each user in the test set are the same, hence gauc == average auc

liuhu-bigeye commented 4 years ago

Thanks for answering