search

Coder-Yu / SELFRec

An open-source framework for self-supervised recommender systems.
516 stars 76 forks source link

关于在yelp2018上复现SimGCL最佳结果的问题 #66

Closed lsc332211 closed 5 months ago

lsc332211 commented 5 months ago

大佬你好，最近在跑对比实验，我按照论文中的最佳参数跑了SimGCL，但是无法达到文中给的结果，相差有点多，不知道是什么问题，求解答。 下面是我的参数和结果： Top 20 Hit Ratio:0.05351 Precision:0.02738 Recall:0.06133 NDCG:0.05087 2024-06-01 16:44:02,744 - SimGCL - INFO - training.set=./dataset/yelp2018/train.txt 2024-06-01 16:44:02,744 - SimGCL - INFO - test.set=./dataset/yelp2018/test.txt 2024-06-01 16:44:02,745 - SimGCL - INFO - model.name=SimGCL 2024-06-01 16:44:02,745 - SimGCL - INFO - model.type=graph 2024-06-01 16:44:02,745 - SimGCL - INFO - item.ranking=-topN 10,20 2024-06-01 16:44:02,745 - SimGCL - INFO - embedding.size=64 2024-06-01 16:44:02,745 - SimGCL - INFO - num.max.epoch=20 2024-06-01 16:44:02,745 - SimGCL - INFO - batch_size=8192 2024-06-01 16:44:02,745 - SimGCL - INFO - learnRate=0.001 2024-06-01 16:44:02,745 - SimGCL - INFO - reg.lambda=0.0001 2024-06-01 16:44:02,745 - SimGCL - INFO - SimGCL=-n_layer 3 -lambda 0.5 -eps 0.1 2024-06-01 16:44:02,745 - SimGCL - INFO - output.setup=-dir ./results/ 2024-06-01 17:43:06,943 - SimGCL - INFO - ###Evaluation Results### 2024-06-01 17:43:06,943 - SimGCL - INFO - ['Top 10\n', 'Hit Ratio:0.03178\n', 'Precision:0.03253\n', 'Recall:0.03679\n', 'NDCG:0.04208\n', 'Top 20\n', 'Hit Ratio:0.05351\n', 'Precision:0.02738\n', 'Recall:0.06133\n', 'NDCG:0.05087\n']

Coder-Yu commented 5 months ago

你这取的是迭代20次完毕后的结果吧。你查看一下中间输出，有个best performance

lsc332211 commented 5 months ago

这个就是best performance，而且还是eopch3的结果，之后就一直下跌到结束，epoch20的结果 ndcg直接跌到了0.03几... 代码我没任何改动，所以很疑惑...

Coder-Yu commented 5 months ago

我check一下 另外我用的batch size是2048

lsc332211 commented 5 months ago

OK 我现在用2048再跑一次试试

Coder-Yu commented 5 months ago
image

刚试了没什么问题呀

lsc332211 commented 5 months ago

目前还没跑完，但是从第二个epoch就一直下跌，rec_loss也和您的差很多 ![Uploading 2.PNG…]()

Coder-Yu commented 5 months ago

看不到你的图呀。你要不重新下载一下再跑跑？可能是哪里改过了？

Coder-Yu commented 5 months ago

一开始是要下跌的，然后会涨回来。

lsc332211 commented 5 months ago

Embedding Dimension: 64 Maximum Epoch: 20 Learning Rate: 0.001 Batch Size: 2048 Regularization Parameter: 0.0001 Specific parameters: n_layer:3 lambda:0.5 eps:0.1
Training Set Size: (user number: 31668, item number 38048, interaction number: 1237259) Test Set Size: (user number: 31668, item number 36073, interaction number: 324147)

Best Performance Epoch: 1, Hit Ratio:0.05024 | Precision:0.02571 | Recall:0.05776 | NDCG:0.04805

Epoch: 2, Hit Ratio:0.04026 | Precision:0.0206 | Recall:0.0464 | NDCG:0.03868

Epoch: 3, Hit Ratio:0.03701 | Precision:0.01894 | Recall:0.0427 | NDCG:0.03564

Epoch: 4, Hit Ratio:0.03582 | Precision:0.01833 | Recall:0.0413 | NDCG:0.03459

Epoch: 5, Hit Ratio:0.03426 | Precision:0.01753 | Recall:0.03945 | NDCG:0.03314

Epoch: 6, Hit Ratio:0.03132 | Precision:0.01603 | Recall:0.03601 | NDCG:0.03033

然后是部分损失函数，图片上传失败，我复制一下

training: 1 batch 100 rec_loss: 0.6928188800811768 cl_loss 3.2166385650634766 training: 1 batch 200 rec_loss: 0.6923909783363342 cl_loss 3.092113733291626 training: 2 batch 100 rec_loss: 0.6896593570709229 cl_loss 2.9386730194091797 training: 2 batch 200 rec_loss: 0.688927173614502 cl_loss 2.928771495819092 training: 6 batch 100 rec_loss: 0.6644822955131531 cl_loss 2.8438878059387207 training: 6 batch 200 rec_loss: 0.6604670882225037 cl_loss 2.8412563800811768

lsc332211 commented 5 months ago

好的，感谢百忙之中回复，我准备待会换一台设备再跑一下，请问learnRate=0.001，reg.lambda=0.0001是没问题的吧？

Coder-Yu commented 5 months ago

嗯，你按Readme上的leaderboard的参数配置肯定是可以跑出来的

lsc332211 commented 5 months ago

感谢大佬，解决了，祝生活愉快。 Top 20 Hit Ratio:0.06398 Precision:0.03274 Recall:0.0728 NDCG:0.05994