About the performence of some baseline

[namenotexist]

Hi, i have tried your model on gowalla dataset, but i found that recall@20 in MF,NMF,gcmc implemented in your code are all about 0.10， which is reported 0.1291, 0.1326, 0.1395 in your paper, what is the reason

[xiangwang1223]

Could you please detail your experimental settings (e.g., learning rate, layer size, embedding size, model depth, and whether pretrained or not)?

Without these details, the results are not that reliable. thanks.

[namenotexist]

gcmc embed_size=64, lr=0.0001, layer_size=[64,64,64], node_dropout=[0.1], mess_dropout=[0.1,0.1,0.1], regs=[1e-5], adj_type=gcmc Best Iter=[18]@[29694.0] recall=[0.11159 0.16493 0.20652 0.23976 0.26751], precision=[0.03519 0.02612 0.02181 0.01905 0.01708], hit=[0.42849 0.53815 0.60697 0.65262 0.68712], ndcg=[0.16498 0.19986 0.22356 0.24146 0.25595]

[namenotexist]

nmf embed_size=64, lr=0.0100, layer_size=[64], keep_prob=[0.9], regs=[1e-5,1e-5,1e-2], Best Iter=[8]@[8244.7] recall=[0.09032 0.14617 0.18829 0.22327 0.25221], precision=[0.02603 0.02144 0.01868 0.01683 0.01538], hit=[0.36710 0.50693 0.58889 0.64529 0.68404], ndcg=[0.11706 0.15456 0.17977 0.19965 0.21561], auc=[0.00000]

[xiangwang1223]

Is this the best hyper-parameter? Have you done a grid search for the hyper-parameter settings?

Please refer to Section 4.2.3 in our paper to get more details about the hyper-parameter settings.

[namenotexist]

i try the most easy model MF, which has the same hyper-perameter setting as Sections 4.2.3，but only has 0.1055 recall@20, which is declared 0.1291 in your paper?

[liuhaifeng0212]

I have the same question in recall@20 , could you release your model best hyper-parameter which used in the paper?

[xiangwang1223]

Please check the python and package version first. Please do the grid search for the best parameter setting for different datasets.

Based on my experiments, the best parameter setting for the gowalla dataset is as follows:

1. MF: embed_size=64, lr=0.0005, regs=[1e-5]
2. NMF: embed_size=64, lr=0.0001, layer_size=[64,64], keep_prob=[0.9,0.9], regs=[1e-3]
3. GC-MC: embed_size=64, lr=0.0005, layer_size=[64], keep_prob=[0.8], regs=[1e-2]

[TedSIWEILIU]

gcmc embed_size=64, lr=0.0001, layer_size=[64,64,64], node_dropout=[0.1], mess_dropout=[0.1,0.1,0.1], regs=[1e-5], adj_type=gcmc Best Iter=[18]@[29694.0] recall=[0.11159 0.16493 0.20652 0.23976 0.26751], precision=[0.03519 0.02612 0.02181 0.01905 0.01708], hit=[0.42849 0.53815 0.60697 0.65262 0.68712], ndcg=[0.16498 0.19986 0.22356 0.24146 0.25595]

Hi, can I ask why we have 5 ndcg/recall/hit etc in a list instead of 1? what is the final result, do we take an average?

[xiangwang1223]

... five results correspond to metrics at different K, i.e., recall@20, @40, @60, @80, and @100. please refer to the hyper-parameter setting --Ks.