Evaluation

[Demirrr]

Hello all,

I have a question pertaining to the reported results.

The equations provided in the appendix of You CAN Teach an Old Dog New Tricks! On Training Knowledge Graph Embeddings show the calculation of MRR. In some works--including Tucker or Hyper, MRR calculated through prediction of objects/tails only given subjects and relations.

However, in DensE, I seem to fail to understand the rationale of implementation of evaluation.

Cheers

[luhaonan20]

Hi,

Thank you for your comments! The evaluation procedure is the same as the code of RotatE ICLR2019.

The logic should be the same as you mentioned in Tucker or Hyper. For example, for the "tail-batch" case during the validation, we fix the heads (subjects) and relation in the test dataset, the tail (subjects) candidate is scoring cross all nodes in the graph. Then we rank the predicted scores and compare with the ground truth candidate to get the MRR, MR, Hits@1, Hits@3, and Hits@10.

[Demirrr]

Dear @luhaonan20,

thank you for your concise answer. To be 100% sure:

Does link prediction performances in Table 1 of DensE correspond link prediction performances in "tail-batch" case or mean of "tail-batch" and "head-batch" cases ?

[luhaonan20]

Dear Demirrr,

It's the mean of "tail-batch" and "head-batch" :)

Cheers

[Demirrr]

Appreciated @luhaonan20