search

nju-websoft / OpenEA

A Benchmarking Study of Embedding-based Entity Alignment for Knowledge Graphs, VLDB 2020
GNU General Public License v3.0
533 stars 81 forks source link
entity-alignment knowledge-graph-embedding

readme

A Benchmarking Study of Embedding-based Entity Alignment for Knowledge Graphs

Contributions Welcome License language-python3 made-with-Tensorflow Paper

Entity alignment seeks to find entities in different knowledge graphs (KGs) that refer to the same real-world object. Recent advancement in KG embedding impels the advent of embedding-based entity alignment, which encodes entities in a continuous embedding space and measures entity similarities based on the learned embeddings. In this paper, we conduct a comprehensive experimental study of this emerging field. This study surveys 23 recent embedding-based entity alignment approaches and categorizes them based on their techniques and characteristics. We further observe that current approaches use different datasets in evaluation, and the degree distributions of entities in these datasets are inconsistent with real KGs. Hence, we propose a new KG sampling algorithm, with which we generate a set of dedicated benchmark datasets with various heterogeneity and distributions for a realistic evaluation. This study also produces an open-source library, which includes 12 representative embedding-based entity alignment approaches. We extensively evaluate these approaches on the generated datasets, to understand their strengths and limitations. Additionally, for several directions that have not been explored in current approaches, we perform exploratory experiments and report our preliminary findings for future studies. The benchmark datasets, open-source library and experimental results are all accessible online and will be duly maintained.

Key contributors ✨

Zequn Sun
Zequn Sun (NJU)
 Wei Hu (NJU)
Wei Hu (NJU)
 Muhao Chen (NJU)
Muhao Chen (UC Davis)
 Haofen Wang (TONGJI)
Haofen Wang (TONGJI)

UPDATE

Table of contents

  1. Library for Embedding-based Entity Alignment
    1. Overview
    2. Getting Started
      1. Code Package Description
      2. Dependencies
      3. Installation
      4. Usage
  2. KG Sampling Method and Datasets
    1. Iterative Degree-based Sampling
    2. Dataset Overview
    3. Dataset Description
  3. Experiment and Results
    1. Experiment Settings
    2. Detailed Results
  4. License
  5. Citation

Library for Embedding-based Entity Alignment

Overview

We use Python and Tensorflow to develop an open-source library, namely OpenEA, for embedding-based entity alignment. The software architecture is illustrated in the following Figure.

The design goals and features of OpenEA include three aspects, i.e., loose coupling, functionality and extensibility, and off-the-shelf solutions.

Getting Started

These instructions cover how to get a copy of the library and how to install and run it on your local machine for development and testing purposes. It also provides an overview of the package structure of the source code.

Package Description

src/
├── openea/
│   ├── approaches/: package of the implementations for existing embedding-based entity alignment approaches
│   ├── models/: package of the implementations for unexplored relationship embedding models
│   ├── modules/: package of the implementations for the framework of embedding module, alignment module, and their interaction
│   ├── expriment/: package of the implementations for evalution methods

Dependencies

Installation

We recommend creating a new conda environment to install and run OpenEA. You should first install tensorflow-gpu (tested on 1.8 and 1.12), graph-tool (tested on 2.27 and 2.29, the latest version would cause a bug), and python-igraph using conda:

conda create --name openea python=3.6 graph-tool==2.40 -c conda-forge
conda activate openea
conda install tensorflow-gpu==1.12
conda install -c conda-forge python-igraph

Then, OpenEA can be installed using pip with the following steps:

git clone https://github.com/nju-websoft/OpenEA.git OpenEA
cd OpenEA
pip install -e .

Usage

The following is an example about how to use OpenEA in Python (We assume that you have already downloaded our datasets and configured the hyperparameters as in the examples.)

import openea as oa

model = oa.kge_model.TransE
args = load_args("hyperparameter file folder")
kgs = read_kgs_from_folder("data folder")
model.set_args(args)
model.set_kgs(kgs)
model.init()
model.run()
model.test()
model.save()

More examples are available here

To run the off-the-shelf approaches on our datasets and reproduce our experiments, change into the ./run/ directory and use the following script:

python main_from_args.py "predefined_arguments" "dataset_name" "split"

For example, if you want to run BootEA on D-W-15K (V1) using the first split, please execute the following script:

python main_from_args.py ./args/bootea_args_15K.json D_W_15K_V1 721_5fold/1/

KG Sampling Method and Datasets

As the current widely-used datasets are quite different from real-world KGs, we present a new dataset sampling algorithm to generate a benchmark dataset for embedding-based entity alignment.

Iterative Degree-based Sampling

The proposed iterative degree-based sampling (IDS) algorithm simultaneously deletes entities in two source KGs with reference alignment until achieving the desired size, meanwhile retaining a similar degree distribution of the sampled dataset as the source KG. The following figure describes the sampling procedure.

Dataset Overview

We choose three well-known KGs as our sources: DBpedia (2016-10),Wikidata (20160801) and YAGO3. Also, we consider two cross-lingual versions of DBpedia: English--French and English--German. We follow the conventions in JAPE and BootEA to generate datasets of two sizes with 15K and 100K entities, using the IDS algorithm:

# Entities Languages Dataset names
15K Cross-lingual EN-FR-15K, EN-DE-15K
15K English D-W-15K, D-Y-15K
100K Cross-lingual EN-FR-100K, EN-DE-100K
100K English-lingual D-W-100K, D-Y-100K

The v1.1 datasets used in this paper can be downloaded from figshare, Dropbox or Baidu Wangpan (password: 9feb). (Note that, we have fixed a minor format issue in YAGO of our v1.0 datasets. Please download our v1.1 datasets from the above links and use this version for evaluation.)

(Recommended) The v2.0 datasets can be downloaded from figshare, Dropbox or Baidu Wangpan (password: nub1).

Dataset Statistics

We generate two versions of datasets for each pair of KGs to be aligned. V1 is generated by directly using the IDS algorithm. For V2, we first randomly delete entities with low degrees (d <= 5) in the source KG to make the average degree doubled, and then execute IDS to fit the new KG. The statistics of the datasets are shown below.

Dataset Description

We hereby take the EN_FR_15K_V1 dataset as an example to introduce the files in each dataset. In the 721_5fold folder, we divide the reference entity alignment into five disjoint folds, each of which accounts for 20% of the total alignment. For each fold, we pick this fold (20%) as training data and leave the remaining (80%) for validation (10%) and testing (70%). The directory structure of each dataset is listed as follows:

EN_FR_15K_V1/
├── attr_triples_1: attribute triples in KG1
├── attr_triples_2: attribute triples in KG2
├── rel_triples_1: relation triples in KG1
├── rel_triples_2: relation triples in KG2
├── ent_links: entity alignment between KG1 and KG2
├── 721_5fold/: entity alignment with test/train/valid (7:2:1) splits
│   ├── 1/: the first fold
│   │   ├── test_links
│   │   ├── train_links
│   │   └── valid_links
│   ├── 2/
│   ├── 3/
│   ├── 4/
│   ├── 5/

Experiment and Results

Experiment Settings

The common hyper-parameters used for OpenEA are shown below.

15K 100K
Batch size for rel. triples 5,000 20,000
Termination condition Early stop when the Hits@1 score begins to drop on
the validation sets, checked every 10 epochs.
Max. epochs 2,000

Besides, it is well-recognized to split a dataset into training, validation and test sets. The details are shown below.

# Ref. alignment # Training # Validation # Test
15K 3,000 1,500 10,500
100K 20,000 10,000 70,000

We use Hits@m (m = 1, 5, 10, 50), mean rank (MR) and mean reciprocal rank (MRR) as the evaluation metrics. Higher Hits@m and MRR scores as well as lower MR scores indicate better performance.

Detailed Results

The detailed and supplementary experimental results are list as follows:

Detailed results of current approaches on the 15K datasets

detailed_results_current_approaches_15K.csv

Detailed results of current approaches on the 100K datasets

detailed_results_current_approaches_100K.csv

Running time (sec.) of current approaches

running_time.csv

Unexplored KG Embedding Models

Detailed results of unexplored KG embedding models on the 15K datasets

detailed_results_unexplored_models_15K.csv

Detailed results of unexplored KG embedding models on the 100K datasets

detailed_results_unexplored_models_100K.csv

License

This project is licensed under the GPL License - see the LICENSE file for details

Citation

If you find the benchmark datasets, the OpenEA library or the experimental results useful, please kindly cite the following paper:

@article{OpenEA,
  author    = {Zequn Sun and
               Qingheng Zhang and
               Wei Hu and
               Chengming Wang and
               Muhao Chen and
               Farahnaz Akrami and
               Chengkai Li},
  title     = {A Benchmarking Study of Embedding-based Entity Alignment for Knowledge Graphs},
  journal   = {Proceedings of the VLDB Endowment},
  volume    = {13},
  number    = {11},
  pages     = {2326--2340},
  year      = {2020},
  url       = {http://www.vldb.org/pvldb/vol13/p2326-sun.pdf}
}

If you use the DBP2.0 dataset, please kindly cite the following paper:

@inproceedings{DBP2,
  author    = {Zequn Sun and
               Muhao Chen and
               Wei Hu},
  title     = {Knowing the No-match: Entity Alignment with Dangling Cases},
  booktitle = {ACL},
  year      = {2021}
}