Multi-Grained Multimodal Interaction Network for Entity Linking

This repository is the official implementation for the paper titled "Multi-Grained Multimodal Interaction Network for Entity Linking".

News

• [2023.09.17] Updating the method of accessing low-resource data and considerations for reproduce the results.
• [2023.09.16] The detailed values of the experimental results have been updated. Please refer here.

Usage

Step 1: Set up the environment

We recommend using Conda to manage virtual environments, and we use Python version 3.8.12.

conda create -n mimic python==3.8.12
conda activate mimic

Please install the specified versions of Python libraries according to the requirements.txt file.

Note that the versions of PyTorch, Transformers, and PyTorch Lightning may have a slight impact on the results. To fully reproduce the results of the paper, we recommend installing the specified versions.

Step 2: Download the data

You may download WikiMEL and RichpediaMEL from https://github.com/seukgcode/MELBench and WikiDiverse from https://github.com/wangxw5/wikiDiverse.

Or download our cleaned data WikiMEL, RichpediaMEL, WikiDiverse (Password: kdd2023).

Step 3: Modify the data path

Please modify the configuration files under the "config" directory (including the YAML files for all 3 datasets) and replace YOUR_PATH in the data field of each configuration file with the path to your corresponding dataset.

NOTE: Due to the uploaded training files of RichpediaMEL, mention images are stored in the folder mention_images. You need to modify the mention_img_folder in the richpediamel.yaml config file or rename the mention_images folder to mention_image. (Thank Zhiwei Hu for bringing up this issue)

Step 4: Start the training

Now you can execute bash run.sh <gpu_id> <dataset_name> to begin the training.

bash run.sh 0 wikimel       # for WikiMEL
bash run.sh 0 richpediamel  # for RichpediaMEL
bash run.sh 0 wikidiverse   # for WikiDiverse

Code Structure

The code is organized as follows:

├── codes
│   ├── main.py
│   ├── model
│   │   ├── lightning_mimic.py
│   │   └── modeling_mimic.py
│   └── utils
│       ├── dataset.py
│       └── functions.py
├── config
│   ├── richpediamel.yaml
│   ├── wikidiverse.yaml
│   └── wikimel.yaml
├── readme.md
├── requirements.txt
└── run.sh

Results

Main Result

Low-resource Setting Result

To access low-resource training data, please refer here.

10% RichpediaMEL

20% RichpediaMEL

10% WikiDiverse

20% WikiDiverse

Citation

If you find this project useful in your research, please cite the following paper:

@inproceedings{luo2023multi,
    author = {Luo, Pengfei and Xu, Tong and Wu, Shiwei and Zhu, Chen and Xu, Linli and Chen, Enhong},
    title = {Multi-Grained Multimodal Interaction Network for Entity Linking},
    year = {2023},
    publisher = {Association for Computing Machinery},
    booktitle = {Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
    pages = {1583–1594},
}

Contact Information

If you have any questions, please contact pfluo@mail.ustc.edu.cn.