How do we calculate AUC for SASRec?

[millenniumbismay]

Hello! Great work and great paper! I am working something related to CTR and Sequential Recommendation. However, I am having a hardtime understanding how the models like SASRec are evaluated on AUC score? SASRec is a Sequential Recommendation where the original paper has metrics like HR@k and NDCG@k. But for CTR prediction like TallRec, how do you get AUC score for SASRec and similar Sequential Recommendors. I tried finding the code for the same. Could you please point me at the piece of code where you calculate AUC for SASRec? Really sorry if I missed.

Thanks and Regards Millennium Bismay

[SAI990323]

I’m sorry, I tried to find this part of the code, but it seems to have been lost in the update after the server crash. Specifically, calling ChatGPT through the API returns the generation probabilities of the top 5 tokens. We extract the probabilities of generating “Yes” and “No” and recalculate their probability values for calculating the AUC. On the other hand, if “Yes” or “No” does not appear in the top 5 tokens, we use 1.0 minus the sum of the generation probabilities of the top 5 tokens as a substitute (which is often in the range of 1e-3 to 1e-5).

[millenniumbismay]

Thank you for your reply! Thank you for mentioning the technique used for caIculating AUC for ChatGPT. However, I am sorry, I think I might have not been very clear about the question I asked earlier.

I wanted to understand how to calculate the AUC for SASRec.

[jizhi-zhang]

Thank you very much for your attention to our work.

We conduct testing in the CTR setting, and AUC is the most classic metric for CTR tasks. This is why we have chosen to use AUC as our evaluation metric.

In terms of implementation, we have transformed the activation function of the last layer of all baselines into the commonly used Sigmoid activation function in CTR tasks. This enables these baselines to be applied to CTR tasks. In this case, the output represents the CTR value of the target item for the user.

Specifically, each recommendation example consists of three parts: (seq, target, label). Here, seq represents the historical interactions, target represents the item to be evaluated, and label indicates whether the user likes the target item, with values of 0 and 1.

Taking SASRec as an example, we input the seq into SASRec and obtain the logits for the corresponding target. Finally, by applying the Sigmoid activation function, we obtain the corresponding CTR value. Since we have labels with values of 0 and 1, we can calculate the AUC to measure the performance of the model in the CTR task.

[swordqiqi]

您好，看了您的回答之后，我依然有些没理解的地方，想请教您。 您是如何定义历史序列的呢？因为对预测目标的分类是喜欢和不喜欢，但在历史序列中并不包含喜欢与不喜欢这样的信息。那您是如何处理的呢？

[millenniumbismay]

Likes and Dislikes are derived from ratings. If I am not wrong, TallRec implements like as rating>3 and rest dislikes. I personally thought ratings >= 3 for like. So its on you how you want to model.

[swordqiqi]

谢谢！！但我想我可能没有表达清楚我的意思，您说的这些我在论文中有注意到。我其实想说的是，对于baseline，您是如何将评分信息加入到历史序列的呢，因为在我的理解中，历史序列是一个曾经和用户发生过交互的物品的序列，但这似乎并不能包含评分的信息，而只是表明用户是否看过这个电影（书籍）。

[millenniumbismay]

So, the data set has two parts - list of historical movie sequence and list of like or dislike as 1 or 0. And hour final prediction basically comes from a sogmoid layer on top of the final layer of SASRec.

On Sat, Apr 13, 2024 at 1:40 AM swordqiqi @.***> wrote:

谢谢！！但我想我可能没有表达清楚我的意思，您说的这些我在论文中有注意到。我其实想说的是，对于baseline，您是如何将评分信息加入到历史序列的呢，因为在我的理解中，历史序列是一个曾经和用户发生过交互的物品的序列，但这似乎并不能包含评分的信息，而只是表明用户是否看过这个电影（书籍）。

— Reply to this email directly, view it on GitHub https://github.com/SAI990323/TALLRec/issues/49#issuecomment-2053529814, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFQQJ5H6XHPD7UO2G5TRHS3Y5DHMVAVCNFSM6AAAAABD62T6AKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJTGUZDSOBRGQ . You are receiving this because you modified the open/close state.Message ID: @.***>

[swordqiqi]

我懂了，谢谢！！Thanks a lot！！！！！