search

Sujit-O / pykg2vec

Python library for knowledge graph embedding and representation learning.
MIT License
602 stars 109 forks source link

Question: performance of QuatE vs RotatE #210

Open dany-nonstop opened 3 years ago

dany-nonstop commented 3 years ago

Issue Template

Performance of QuatE is not comparable to that of RotateE

Describe the bug I tried to reproduce performance of both QuatE and RotateE

To Reproduce Steps to reproduce the behavior:

  1. Run these two commands 
    pykg2vec-train -exp True -mn rotate -ds freebase15k -device cuda
    pykg2vec-train -exp True -mn quate-ds freebase15k -device cuda
  2. Performance reported for RotatE 
    ------Test Results for freebase15k: Epoch: 320 --- time: 1372.47------------
--# of entities, # of relations: 14951, 1345
--mr,  filtered mr             : 223.5828, 62.3827
--mrr, filtered mrr            : 0.2553, 0.7473
--hits1                        : 0.1424 
--filtered hits1               : 0.6784 
--hits3                        : 0.2839 
--filtered hits3               : 0.7953 
--hits5                        : 0.3800 
--filtered hits5               : 0.8253 
--hits10                        : 0.5106 
--filtered hits10               : 0.8598

    for QuatE

    ------Test Results for freebase15k: Epoch: 410 --- time: 511.22------------
--# of entities, # of relations: 14951, 1345
--mr,  filtered mr             : 243.7488, 84.9533
--mrr, filtered mrr            : 0.2333, 0.6890
--hits1                        : 0.1218 
--filtered hits1               : 0.6097 
--hits3                        : 0.2576 
--filtered hits3               : 0.7409 
--hits5                        : 0.3542 
--filtered hits5               : 0.7797 
--hits10                        : 0.4899 
--filtered hits10               : 0.8239

Expected behavior My impression is that QuatE as an extension to RotatE, should perform better than the latter. In my experiments, RotatE's performance is pretty close to reported by its authors and QuatE authors, whereas QuatE itself is a far cry from its authors' claim.

Desktop (please complete the following information):