mnqu
commented
1 year ago We have updated the codes, and you could now reproduce the results easily : )
Closed navdeepkjohal closed 1 year ago
mnqu
commented
1 year ago We have updated the codes, and you could now reproduce the results easily : )
Hi,
We are students from IIT Delhi who were thinking about building on your model for our project. After working with the code, we have been able to reproduce the results of RNNLogic with embedding on WN18RR dataset.
However, despite our best efforts, we have not been able to reproduce the results for RNNLogic+ w/o embeddings and RNNLogic+ with embeddings for WN18RR dataset. The best results that we could obtained so far for RNNLogic+ w/o embeddings for WN18RR dataset are:
MR: 7301.43, MRR:0.467, Hits@1: 0.438, Hits@3: 0.48, Hits@10: 0.52
compared to:
MR: 7204, MRR: 0.489, Hits@1: 0.453 Hits@3: 0.506 Hits@10: 0.563
reported in the paper. Further, the best results we have been able to obtain for RNNLogic+ with embeddings are:
MR: 4738.723, MRR: 0.474, Hits@1: 0.436, Hits@3: 0.489, Hits@10: 0.551
compared to:
MR: 4624, MRR: 0.513, Hits@1: 0.471 Hits@3: 0.532 Hits@10: 0.597
reported in the paper. We would also like to mention that we produced our best results with settings that are very different from the ones mentioned in the paper, for instance, we set eta in equation (11) to 5.0 to obtain the results for RNNLogic+ with embeddings on WN18RR in our case as opposed to the optimal eta of 0.5 reported in the paper. We trained RNNLogic to generate rules of length 3, instead of rules of length 5 as mentioned in the paper. In addition, the RNNLogic implementation in the new version of the code performs far worse than the old version of the code, and is missing several components such as the pseudogroundings and RotatE embeddings. Thus, we used the old code to train RNNLogic and new code to train RNNLogic+.
It would be really helpful if you could release the exact settings under which you ran your model to produce the results reported in the paper. This would also include the settings of certain flags like 'init_weight_boot', 'use_neg_rules' etc. as the model seems to be particularly sensitive to these hyper-parameters. Even better, if you could please release the best trained models for RNNLogic (the model_r.pth files in the codes/workspace folder) with the hyperparameter setting that gave best results for you, it would help us build our model on top of that.
We are looking forward to a favorable response from you!
Regards, Ananjan Nandi Navdeep Kaur