PyTorch Implementation of KPRN

Introduction

This repository is Pytorch implementation of Knowledge-guided Pairwise Reconstruction Network for Weakly Supervised Referring Expression Grounding in ACM MM 2019. Check our paper for more details.

Prerequisites

Python 3.5
Pytorch 0.4.1
CUDA 8.0

Installation

Please refer to MattNet to install mask-faster-rcnn, REFER and refer-parser2. Follow Step 1 & 2 in Training to prepare the data and features.
Calculate semantic similarity as supervision infotmation.

Download Glove word embedding.

cache/word_embedding/download_embed_matrix.sh

Generate semantic similarity and word embedding file.

python tools/prepro_sub_obj_wds.py --dataset ${DATASET} --splitBy ${SPLITBY}
python tools/prepro_sim.py --dataset ${DATASET} --splitBy ${SPLITBY}

Training

Train KPRN with ground-truth annotation:

CUDA_VISIBLE_DEVICES=${GPU_ID} python ./tools/train.py --dataset ${DATASET} --splitBy ${SPLITBY} --exp_id ${EXP_ID} --sub_filter_type ${SUBJECT_FILTER_TYPE} --sub_filter_thr ${SUBJECT_FILTER_THRESHOLD}

Evaluation

Evaluate KPRN with ground-truth annotation:

CUDA_VISIBLE_DEVICES=${GPU_ID} python ./tools/eval.py --dataset ${DATASET} --splitBy ${SPLITBY} --split ${SPLIT} --id ${EXP_ID}

Citation

@inproceedings{lxj2019kprn,
  title={Knowledge-guided Pairwise Reconstruction Network for Weakly Supervised Referring Expression Grounding},
  author={Xuejing Liu, Liang Li, Shuhui Wang, Zheng-Jun Zha, Li Su, and Qingming Huang},
  booktitle={ACM MM},
  year={2019}
}

Acknowledgement

Thanks for the work of Licheng Yu. Our code is based on the implementation of MattNet.

Authorship

This project is maintained by Xuejing Liu.

GingL / KPRN

readme