Closed qibinc closed 5 years ago
Hi Qibin,
Thank you for your interest in this work!
Sorry for the confusion, indeed an error has slipped in this table and we updated our paper on Arxiv a few weeks ago https://arxiv.org/abs/1812.07617
In this line, I modified data="db_pretrain" to data="db". To the best of my knowledge, this is sufficient to produce the No pre-training result on REDIAL.
That is correct
As for the hyper-parameters, we used the following in the experiments on ReDial without pre-training:
autorec_params = {
'layer_sizes': [100],
'f': "sigmoid",
'g': "sigmoid",
}
These should produce the results in table 2 of the updated paper on Arxiv.
The hyper-parameters in test_params.py
correspond to those we used to train the complete conversational recommender system. In the future I'll try to add which hyper-parameters we used to produce the results on the different tables.
Finally, I would just like to mention that this table only aims to justify our approach that consists in pre-training the recommender system on another dataset, and I am sure there are more intelligent ways to evaluate a recommender system
Hope this helped!
Hi Raymond,
Thanks for your help!
Thanks for sharing the new link. I should have check the latest version first.. BTW, I suggest that you also change the paper link in this repo's README to the arxiv link.
Finally, I would just like to mention that this table only aims to justify our approach that consists in pre-training the recommender system on another dataset, and I am sure there are more intelligent ways to evaluate a recommender system
Thanks for mention it. I understand that you used RMSE to demonstrate straightforwardly the effectiveness of pretraining and also to provide some insights on the data. I'll definitely try to adopt a more common evaluation.
You can close the issue.
Best wishes!
Thanks for catching that ;) I updated the link
Hi Raymond, @RaymondLi0
This work is super interesting. Congrads to you that it got into NeurIPS 2018!
My name is Qibin, from Tsinghua University, China. I'm currently trying to follow up this work and have started to playing with your baselines. However, I cannot reproduce some part of the results in
Table 2: RMSE for movie recommendations
.Here are the details:
For
Experiments on REDIAL (validation RMSE), No pre-training on MovieLens
, the best validation loss I got is 0.0756 (RMSE = sqrt(0.0756) = 0.275) while the result in the paper is 0.127. There is a large gap in there and I have no idea what is wrong. In particular, could you check the hyperparameters in the repo?Here are the changes I made to the code:
In this line, I modified
data="db_pretrain"
todata="db"
. To the best of my knowledge, this is sufficient to produce theNo pre-training
result on REDIAL.Looking forward to your reply. Thanks!
In the end, I provide my training output.