search

hwwang55 / PathCon

Combining relational context and relational paths for knowledge graph completion
MIT License
131 stars 29 forks source link

Results reproducibility #2

Closed vitornl closed 4 years ago

vitornl commented 4 years ago

Hi,

I'm trying to reproduce the results presented in the paper using the WN18RR dataset. However, I'm running into some kind of trouble since I'm not able to replicate the results. Using the default parameters, i.e., running only python main.py the obtained results are, for example:

final results
acc: 1.0000   mrr: 0.4864   mr: 3.9359   h1: 0.3086   h3: 0.5443   h5: 0.7096

What I'm doing wrong?

Also, the other models' differ greatly from everything reported so far, as an example, the RotatE results for the WN18RR dataset reported in the original paper are mmr: 0.440 h1: 36.1 h3: 48.3 h10: 58.1, while in the PathCon paper are mmr: 0.799 h1: 0.735 h3: 0.823. How do they were evaluated?

Thanks in advance.

hwwang55 commented 4 years ago

Hi! For your first question, the result is quite weird since we never get a result of accuracy 1.0. I just ran the code again and below is the result:

image image image

Could you please post the logs printed during training just as above? For your second question, the result reported in our paper is for relation prediction (given h and t, predict r) while in previous papers they do tail or head prediction (given h and r, predict t, or given t and r, predict h).

vitornl commented 4 years ago

Thanks for the answer. I was able to reproduce after re-cloning the repo. I think I ran into some problems with the data and the generation process.