search

dfdazac / blp

"Inductive Entity Representations from Text via Link Prediction" @ The Web Conference 2021
MIT License
58 stars 6 forks source link

why FB15k-237 test and train entities not disjoin? #7

Closed mx8435 closed 2 years ago

mx8435 commented 2 years ago

The entity "/m/01by1l" exists both in 'ind-test.tsv' and 'ind-train.tsv'. @dfdazac

image

dfdazac commented 2 years ago

Hi, thanks for reaching out. In our original publication we refer to two types of evaluation: dynamic and transfer.

In the dynamic evaluation, we model the scenario when new entities are added to an existing graph. This means that triples in the test set can contain entities already in the training set. We carry out this evaluation with the WN18RR and FB15k-237 datasets.

In the transfer scenario, all entities in the test set are assumed to be new, and at test time, link prediction is evaluated considering as candidates only the set of entities in the test set. We show this with the Wikidata5M dataset.

You might find additional information of interest in appendix A of our paper.

Please let me know if this helps with your question.

sanxing-chen commented 1 year ago

If the validation and test can contain entities in the training set, the statistics in Table 2, showing that 4094 and 1454 entities are in the validation and test set of WN18RR and FB15k-237 respectively, would be incorrect.