Data leakage?

[GLA1VE11]

Dear Author: Hello! I recently read the paper and found a very serious problem: there is a very serious data leakage in the paper. In the testing phase, due to the augmentation of the data, one original session is split into many sessions (e.g. splitting 1-2-3-4->5 into 1->2; 1-2->3; 1-2-3->4), but since there is no shuffle operation in the paper, it results in predicting 1-2->3 when there is already a 1-2-3 edge, which would greatly improve the performance of the model. Here are the results I got using the original code (yoochoose64 dataset， after 3 epoch): Recall@20: 82.5182 MRR@20: 50.5478 Epoch: 2,2 Here are the results after I used the shuffle operation(after 5 epoch)： Recall@20: 71.2745 MRR@20: 30.8134 Epoch: 2,2

[YuhanZhen]

Thank you very much for your information. We also noticed this problem, so we switched the batch split method to random and also changed the batch size for evaluation in the local device. The performance does not decline notably. By the way, we have corrected this error in our code. Thanks again for your notification. I will appreciate it if you could run more times.

[GLA1VE11]

Thank you very much for your information. We also noticed this problem, so we switched the batch split method to random and also changed the batch size for evaluation in the local device. The performance does not decline notably. By the way, we have corrected this error in our code. Thanks again for your notification. I will appreciate it if you could run more times.

Thanks for your quick reply! But I found that the statement on line 58 in utils.py is commented out, so just changing shuffle=True will not work. I tried yoochoose1_64 again locally and found that the model does converge around the 7th epoch with the following results: Recall@20: 70.9471 MRR@20: 30.7142 Epoch: 6, 6