search

mquad / hgru4rec

Code for our ACM RecSys 2017 paper "Personalizing Session-based Recommendation with Hierarchical Recurrent Neural Networks"
MIT License
215 stars 71 forks source link

About the data statistics and performance #7

Closed gtshs2 closed 5 years ago

gtshs2 commented 6 years ago

First, thank you for sharing a good paper and code.

We have inquiries about data and performance.

  1. data statistics When xing data was preprocessed based on your code, 58,035 items were derived. The number of users and the number of events are the same as mentioned in your paper, but the number of items is different. I wonder if you can recheck the number of items.

  2. Performance When I experiment through your scripts, I couldn't get as much performance as in your paper. When I run the model with ten epochs, Recall@5 values of HRNN-All(#hidden=100) were 0.1280 and HRNN-All(#hidden=500) was 0.1355. (the performance were 0.1334 and 0.1482 respectively in your paper) Increasing the epoch did not change the outcome significantly. Please check if the script you uploaded is correct for getting the best performance.

mquad commented 6 years ago

Hi, thanks for notifying the issues. I will take a closer look as soon as possible. Thanks for the patience

mquad commented 6 years ago

Hi @gtshs2 , I just found some time to check the aforementioned issues. Regarding 1., I don't see any difference in the data statistics. The number of items is exactly the same as reported in the paper. Here is the output of the data generation script: Building sessions Original data: Num items: 1002161 Num users: 770859 Num sessions: 2260912 Filtering data Filtered data: Num items: 59297 Num users: 11479 Num sessions: 89755 Partitioning data Write to disk

Are you using the correct interactions.csv file from the XING 2016 challenge?

It's highly likely the differences in the performance highlighted in 2. are due to the differences in the training dataset.

mquad commented 5 years ago

FYI, evaluation.py contained an error (rank_ok = ranks < cut_off instead of rank_ok = ranks <= cut_off) which is now fixed. This ensures the metrics will be the same as in the paper.